Sep 09 20:09:15.445157 ip-10-0-131-185 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:09:15.445169 ip-10-0-131-185 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:09:15.445176 ip-10-0-131-185 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:09:15.445475 ip-10-0-131-185 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:09:25.247567 ip-10-0-131-185 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:09:25.247583 ip-10-0-131-185 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9363a08e2080459dabde65e80ae077d5 --
Sep 09 20:11:14.857411 ip-10-0-131-185 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:11:15.279757 ip-10-0-131-185 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:15.279757 ip-10-0-131-185 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:11:15.279757 ip-10-0-131-185 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:15.279757 ip-10-0-131-185 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:11:15.279757 ip-10-0-131-185 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:15.280768 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.280681    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:11:15.286532 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286511    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:15.286532 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286530    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:15.286532 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286535    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:15.286532 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286538    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286541    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286544    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286547    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286549    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286552    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286555    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286557    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286560    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286563    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286565    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286568    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286570    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286573    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286575    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286578    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286580    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286584    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286588    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286591    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:15.286705 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286593    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286596    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286601    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286604    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286607    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286609    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286612    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286614    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286617    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286620    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286622    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286624    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286627    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286630    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286633    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286635    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286639    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286642    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286644    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286647    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:15.287192 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286649    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286652    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286654    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286656    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286659    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286661    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286664    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286666    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286669    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286671    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286674    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286676    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286679    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286681    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286684    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286686    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286688    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286691    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286693    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286695    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:15.287674 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286698    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286700    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286702    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286705    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286710    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286714    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286718    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286721    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286724    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286726    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286729    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286731    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286734    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286737    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286741    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286744    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286746    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286749    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:15.288202 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286751    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286754    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286762    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286764    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286767    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286769    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286771    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286774    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286776    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286779    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286781    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286784    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286786    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286788    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286791    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286793    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286795    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.286798    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287507    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287513    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:15.288637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287518    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287522    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287527    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287530    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287533    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287536    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287538    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287541    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287543    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287546    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287548    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287551    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287553    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287556    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287558    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287562    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287565    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287567    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287570    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:15.289149 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287572    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287575    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287578    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287580    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287582    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287585    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287587    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287589    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287592    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287594    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287596    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287599    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287601    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287604    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287607    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287610    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287612    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287615    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287617    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287619    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:15.289620 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287622    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287624    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287627    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287630    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287634    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287636    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287639    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287641    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287644    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287647    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287650    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287652    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287655    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287657    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287660    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287662    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287665    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287667    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287669    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287672    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:15.290122 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287674    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287677    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287681    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287685    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287688    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287692    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287695    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287698    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287701    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287704    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287707    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287709    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287712    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287714    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287717    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287719    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287722    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287724    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287727    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:15.290605 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287730    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287732    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287735    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287737    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287740    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287743    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287745    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287748    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287750    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287753    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287756    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287759    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287761    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287764    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287767    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287769    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287772    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287774    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.287777    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288166    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:15.291068 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288172    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288175    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288178    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288180    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288183    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288186    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288189    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288191    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288194    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288196    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288199    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288201    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288204    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288207    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288210    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288213    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288216    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288218    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288221    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288223    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:15.291577 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288226    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288229    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288231    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288234    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288236    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288239    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288242    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288246    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288250    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288252    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288255    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288257    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288260    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288263    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288266    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288268    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288271    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288273    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288277    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:15.292064 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288280    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288283    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288286    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288289    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288291    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288294    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288297    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288299    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288302    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288305    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288307    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288310    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288313    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288315    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288317    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288320    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288323    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288325    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288328    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288331    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:15.292534 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288333    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288336    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288338    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288340    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288343    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288345    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288348    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288350    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288352    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288355    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288357    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288359    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288362    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288364    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288367    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288369    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288371    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288374    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288376    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288378    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:15.293057 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288381    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288383    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288386    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288389    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288391    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288396    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288400    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288403    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288406    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288408    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288412    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288415    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288417    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288420    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288422    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288425    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288428    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288430    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.288432    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288513    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:11:15.293576 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288560    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288568    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288572    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288581    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288584    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288588    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288592    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288595    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288598    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288601    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288604    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288607    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288610    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288613    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288615    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288620    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288622    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288628    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288630    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288634    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288636    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288640    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288646    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:11:15.294057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288649    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288652    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288655    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288658    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288661    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288663    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288666    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288669    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288673    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288676    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288678    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288681    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288685    2566 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288687    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288692    2566 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288695    2566 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288698    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288701    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288704    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288708    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288710    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288713    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288716    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288719    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288721    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:11:15.294599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288724    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288728    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288732    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288735    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288737    2566 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288741    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288745    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288748    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288751    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288754    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288758    2566 flags.go:64] FLAG: --help="false"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288760    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288763    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288766    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288769    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288772    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288775    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288778    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288780    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288783    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288786    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288789    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288791    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288794    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:11:15.295235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288797    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288799    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288802    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288805    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288807    2566 flags.go:64] FLAG: --lock-file=""
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288810    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288813    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288816    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288821    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288825    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288830    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288833    2566 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288835    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288838    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288842    2566 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288845    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288850    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288853    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288857    2566 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288859    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288862    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288865    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288867    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288870    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288873    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:11:15.295823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288876    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288883    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288886    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288889    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288892    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288894    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288900    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288903    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288906    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288908    2566 flags.go:64] FLAG: --port="10250"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288911    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288914    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-017e9af8deab0d7e4"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288917    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288920    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288923    2566 flags.go:64] FLAG: --register-node="true"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288925    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288933    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288938    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288941    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288943    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288947    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288950    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288953    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288956    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288959    2566 flags.go:64] FLAG: --runonce="false"
Sep 09 20:11:15.296483 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288962    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288964    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288967    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288970    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288972    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288975    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288979    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288982    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288984    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288987    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288989    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288992    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288995    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.288998    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289000    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289005    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289008    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289011    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289015    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289017    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289020    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289022    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289025    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289028    2566 flags.go:64] FLAG: --v="2"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289033    2566 flags.go:64] FLAG: --version="false"
Sep 09 20:11:15.297144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289038    2566 flags.go:64] FLAG: --vmodule=""
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289041    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.289044    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289187    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289193    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289196    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289199    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289202    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289204    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289207    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289209    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289212    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289215    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289218    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289220    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289222    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289225    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289228    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289230    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289233    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:15.297760 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289235    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289238    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289240    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289243    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289246    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289248    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289250    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289253    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289255    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289258    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289260    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289265    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289268    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289273    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289276    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289280    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289285    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289288    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289291    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289294    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:15.298260 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289296    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289299    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289302    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289305    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289307    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289309    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289312    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289315    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289317    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289320    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289323    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289325    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289328    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289331    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289334    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289337    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289340    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289342    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289344    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289347    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:15.298753 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289349    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289352    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289355    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289358    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289361    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289363    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289366    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289369    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289373    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289376    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289379    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289383    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289386    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289389    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289391    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289394    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289397    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289399    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289402    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:15.299254 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289404    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289407    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289409    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289412    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289414    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289416    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289419    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289421    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289424    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289426    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289429    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289431    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289433    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289436    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289438    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289441    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289445    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289447    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289449    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289452    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:15.299804 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289455    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289459    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.289461    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.290241    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.298835    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.298850    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.299785    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.299796    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299854    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299859    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299864    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299868    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299871    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299874    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:15.300347 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299877    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299880    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299882    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299885    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299887    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299890    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299894    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299897    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299900    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299902    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299905    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299907    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299910    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299913    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299915    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299917    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299920    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299922    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299925    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299927    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:15.300695 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299930    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299932    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299935    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299939    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299942    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299944    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299947    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299949    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299952    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299954    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299957    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299960    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299962    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299964    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299967    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299969    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299972    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299974    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299977    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299979    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:15.301198 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299982    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299984    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299987    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299989    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299991    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299994    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.299997    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300000    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300002    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300004    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300007    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300009    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300011    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300014    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300016    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300019    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300022    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300025    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300027    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300030    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:15.301682 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300032    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300035    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300038    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300040    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300043    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300045    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300047    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300050    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300052    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300054    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300057    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300059    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300062    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300064    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300067    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300069    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300071    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300074    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300076    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300097    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:15.302174 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300100    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300102    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300105    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300107    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300110    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300112    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300115    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300118    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300122    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300127    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300130    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300133    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300135    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.300141    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300252    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:15.302654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300257    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300260    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300262    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300265    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300268    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300270    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300273    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300276    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300278    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300281    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300283    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300287    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300291    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300294    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300297    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300299    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300303    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300307    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300309    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:15.303020 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300312    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300314    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300316    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300319    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300321    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300323    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300326    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300329    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300332    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300335    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300337    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300340    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300342    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300345    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300347    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300350    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300353    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300356    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300358    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300361    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:15.303492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300364    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300366    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300369    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300371    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300374    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300376    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300378    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300381    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300383    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300385    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300388    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300390    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300393    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300395    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300397    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300400    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300402    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300405    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300408    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300410    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:15.303969 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300413    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300416    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300418    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300420    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300423    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300425    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300427    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300430    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300432    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300436    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300439    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300441    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300444    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300447    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300449    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300452    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300454    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300456    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300459    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300461    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:15.304464 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300464    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300466    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300468    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300471    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300473    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300476    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300478    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300480    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300483    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300485    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300488    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300490    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300493    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300495    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300498    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300500    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300502    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300505    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:15.304942 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:15.300507    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:15.305428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.300512    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:15.305428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.300617    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:11:15.305428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.303036    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:11:15.305428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.304185    2566 server.go:1019] "Starting client certificate rotation"
Sep 09 20:11:15.305428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.304283    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:15.305428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.304320    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:15.307746 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.307732    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:15.313277 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.313255    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:15.321300 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.321284    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:11:15.324639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.324625    2566 log.go:25] "Validated CRI v1 image API"
Sep 09 20:11:15.326228 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.326204    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:11:15.330014 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.329989    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c05e423a-932f-40ed-a95b-64e803ca4659:/dev/nvme0n1p3 d917776a-c2cb-4c00-8d67-75e2033cfe76:/dev/nvme0n1p4]
Sep 09 20:11:15.330109 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.330013    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:11:15.335284 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.335162    2566 manager.go:216] Machine: {Timestamp:2026-09-09 20:11:15.333636384 +0000 UTC m=+0.369107535 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104027 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0575e7fd0feb5cd483cee86b1e5e SystemUUID:ec2e0575-e7fd-0feb-5cd4-83cee86b1e5e BootID:9363a08e-2080-459d-abde-65e80ae077d5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:08:63:a6:19:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:08:63:a6:19:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:74:bf:1e:2c:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:11:15.335284 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.335275    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:11:15.335440 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.335354    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:11:15.335440 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.335399    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:15.339139 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.339110    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:11:15.339271 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.339139    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:11:15.339934 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.339925    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:11:15.339966 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.339937    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:11:15.339966 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.339946    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:15.339966 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.339959    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:15.341853 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.341843    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:11:15.345433 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345421    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:15.345552 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345542    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:11:15.345590 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345574    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:11:15.345590 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345584    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:11:15.345648 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345599    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:11:15.345648 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345608    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:11:15.345648 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345617    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:11:15.345975 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345962    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:15.346027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.345984    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:15.348416 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.348399    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:11:15.351508 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351492    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:11:15.351579 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351523    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:11:15.351723 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351710    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:11:15.351765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351729    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:11:15.351765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351735    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:11:15.351765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351741    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:11:15.351765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351749    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:11:15.351765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.351754    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:11:15.352377 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.352367    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:11:15.352377 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.352376    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:11:15.352442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.352383    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:11:15.352442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.352389    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:11:15.352442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.352397    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:11:15.352442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.352405    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:11:15.354282 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.354268    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:11:15.354322 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.354285    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:11:15.357330 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.357300    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:11:15.357431 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.357319    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:11:15.358365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.358350    2566 server.go:1268] "Started kubelet"
Sep 09 20:11:15.359024 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.358986    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:11:15.359208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.359150    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:11:15.359292 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.359275    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:11:15.361053 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.361003    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:11:15.361890 ip-10-0-131-185 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:11:15.362765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.362617    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:11:15.368273 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.368236    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2cmz"
Sep 09 20:11:15.368605 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.368581    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:11:15.370362 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.370341    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:11:15.371460 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.370293    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-185.ec2.internal.18d3c0157359a9f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-185.ec2.internal,UID:ip-10-0-131-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-185.ec2.internal,},FirstTimestamp:2026-09-09 20:11:15.358321137 +0000 UTC m=+0.393792288,LastTimestamp:2026-09-09 20:11:15.358321137 +0000 UTC m=+0.393792288,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-185.ec2.internal,}"
Sep 09 20:11:15.373497 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.373470    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:15.373897 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.373874    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h2cmz"
Sep 09 20:11:15.374108 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.374074    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:11:15.374296 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.374273    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:15.374998 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.374983    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:11:15.375148 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375130    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:11:15.375148 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375150    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:11:15.375261 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375239    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:11:15.375321 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375249    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:11:15.375623 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375605    2566 factory.go:55] Registering systemd factory
Sep 09 20:11:15.375696 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375653    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:11:15.375888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375875    2566 factory.go:153] Registering CRI-O factory
Sep 09 20:11:15.375888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375887    2566 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:11:15.376063 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375930    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:11:15.376063 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375947    2566 factory.go:103] Registering Raw factory
Sep 09 20:11:15.376063 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.375959    2566 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:11:15.376282 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.376265    2566 manager.go:349] Starting recovery of all containers
Sep 09 20:11:15.379217 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.379197    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:15.383247 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.383220    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-185.ec2.internal\" not found" node="ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.385857 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.385841    2566 manager.go:354] Recovery completed
Sep 09 20:11:15.390410 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.390396    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:15.392434 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.392416    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:15.392496 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.392447    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:15.392496 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.392457    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:15.393071 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.393058    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:11:15.393171 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.393072    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:11:15.393171 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.393110    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:15.393268 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.393198    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:15.393862 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.393848    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:15.393926 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.393880    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:15.393926 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.393895    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:15.394331 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.394320    2566 policy_none.go:49] "None policy: Start"
Sep 09 20:11:15.394374 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.394336    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:11:15.394374 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.394345    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:11:15.395228 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.395217    2566 policy_none.go:47] "Start"
Sep 09 20:11:15.442671 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.442651    2566 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.442698    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.442711    2566 server.go:87] "Starting device plugin registration server"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.443143    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.443165    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.443253    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.443374    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.443384    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.443956    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:11:15.452574 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.444008    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:15.483536 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.483505    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:11:15.484760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.484738    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:11:15.484858 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.484791    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:11:15.484858 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.484817    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:11:15.484858 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.484858    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:11:15.487245 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.487225    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:15.543511 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.543414    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:15.544406 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.544388    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:15.544469 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.544424    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:15.544469 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.544436    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:15.544469 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.544456    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.552155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.552139    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.552155 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.552161    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-185.ec2.internal\": node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:15.566305 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.566283    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:15.585438 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.585416    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal"]
Sep 09 20:11:15.585541 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.585509    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:15.587286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.587271    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:15.587366 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.587300    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:15.587366 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.587310    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:15.589651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.589639    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:15.589794 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.589776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.589833 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.589805    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:15.590378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.590364    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:15.590451 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.590388    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:15.590451 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.590402    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:15.590451 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.590436    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:15.590560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.590455    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:15.590560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.590464    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:15.592647 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.592634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.592687 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.592660    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:15.593351 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.593336    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:15.593403 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.593363    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:15.593403 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.593376    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:15.621791 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.621766    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-185.ec2.internal\" not found" node="ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.625944 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.625927    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-185.ec2.internal\" not found" node="ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.666578 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.666550    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:15.676325 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.676298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b0eae04d84f726f4b8e2b038df85374-config\") pod \"kube-apiserver-proxy-ip-10-0-131-185.ec2.internal\" (UID: \"7b0eae04d84f726f4b8e2b038df85374\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.676325 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.676323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b89797f28a6a6731f98f222654fb2c7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal\" (UID: \"b89797f28a6a6731f98f222654fb2c7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.676470 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.676346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89797f28a6a6731f98f222654fb2c7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal\" (UID: \"b89797f28a6a6731f98f222654fb2c7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.766929 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.766893    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:15.777282 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.777252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b89797f28a6a6731f98f222654fb2c7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal\" (UID: \"b89797f28a6a6731f98f222654fb2c7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.777350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.777271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b89797f28a6a6731f98f222654fb2c7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal\" (UID: \"b89797f28a6a6731f98f222654fb2c7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.777350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.777303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89797f28a6a6731f98f222654fb2c7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal\" (UID: \"b89797f28a6a6731f98f222654fb2c7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.777418 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.777354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b0eae04d84f726f4b8e2b038df85374-config\") pod \"kube-apiserver-proxy-ip-10-0-131-185.ec2.internal\" (UID: \"7b0eae04d84f726f4b8e2b038df85374\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.777418 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.777386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7b0eae04d84f726f4b8e2b038df85374-config\") pod \"kube-apiserver-proxy-ip-10-0-131-185.ec2.internal\" (UID: \"7b0eae04d84f726f4b8e2b038df85374\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.777418 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.777404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89797f28a6a6731f98f222654fb2c7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal\" (UID: \"b89797f28a6a6731f98f222654fb2c7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.867675 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.867606    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:15.924123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.924096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.928253 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:15.928230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:15.967854 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:15.967819    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:16.068378 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:16.068349    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:16.168983 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:16.168912    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:16.269532 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:16.269499    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:16.305099 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.305061    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:11:16.305484 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.305260    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:16.305484 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.305272    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:16.370558 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:16.370517    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:16.376522 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.376487    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:06:15 +0000 UTC" deadline="2028-06-12 23:22:53.61194937 +0000 UTC"
Sep 09 20:11:16.376522 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.376519    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15411h11m37.235433453s"
Sep 09 20:11:16.471508 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:16.471443    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-185.ec2.internal\" not found"
Sep 09 20:11:16.517500 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:16.517459    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89797f28a6a6731f98f222654fb2c7f.slice/crio-19b41b49ad70d1575152fe1e4ddb5c577ad795f48ce5a59f19f456958157eaed WatchSource:0}: Error finding container 19b41b49ad70d1575152fe1e4ddb5c577ad795f48ce5a59f19f456958157eaed: Status 404 returned error can't find the container with id 19b41b49ad70d1575152fe1e4ddb5c577ad795f48ce5a59f19f456958157eaed
Sep 09 20:11:16.518042 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:16.518020    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0eae04d84f726f4b8e2b038df85374.slice/crio-28d3221bd988ccffb5c9459f1d5dd7c66be4ad1bcde7a90865f4ec43095e4542 WatchSource:0}: Error finding container 28d3221bd988ccffb5c9459f1d5dd7c66be4ad1bcde7a90865f4ec43095e4542: Status 404 returned error can't find the container with id 28d3221bd988ccffb5c9459f1d5dd7c66be4ad1bcde7a90865f4ec43095e4542
Sep 09 20:11:16.523022 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.523004    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:11:16.524057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.524042    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:16.574426 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.574401    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:16.585991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.585972    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:16.587761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.587747    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal"
Sep 09 20:11:16.596151 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.596126    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:16.617069 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:16.617043    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:17.346750 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.346700    2566 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:17.355821 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.355510    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:17.357394 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.357363    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal","openshift-multus/multus-additional-cni-plugins-ddscx","openshift-multus/multus-fk4th","openshift-cluster-node-tuning-operator/tuned-tmptz","openshift-image-registry/node-ca-rzvkf","openshift-multus/network-metrics-daemon-qhxc4","openshift-network-diagnostics/network-check-target-2vsjc","openshift-network-operator/iptables-alerter-w4lv4","openshift-ovn-kubernetes/ovnkube-node-h7z6h","kube-system/global-pull-secret-syncer-tq7t4","kube-system/konnectivity-agent-tvmd6","kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf","openshift-dns/node-resolver-pt8r9"]
Sep 09 20:11:17.360099 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.360074    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:17.360223 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.360197    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:17.362618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.362598    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.364970 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.364941    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.365274 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.365252    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:17.365578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.365562    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:17.365651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.365619    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:17.365726 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.365712    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b946f\""
Sep 09 20:11:17.367366 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.367340    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.367543 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.367528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dbjdx\""
Sep 09 20:11:17.367899 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.367844    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:17.367989 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.367925    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:17.368283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.368261    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:17.368579 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.368562    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:17.369917 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.369899    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.370239 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.370219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:17.370343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.370314    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:17.370417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.370361    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kqgk\""
Sep 09 20:11:17.372585 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.372570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:17.372838 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.372667    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:17.373270 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.373247    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9vskd\""
Sep 09 20:11:17.373351 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.373292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:17.373394 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.373370    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:17.373394 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.373248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:17.374717 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.374698    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:17.375092 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.375073    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.377710 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.377689    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m6f54\""
Sep 09 20:11:17.377779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.377723    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:17.377911 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.377895    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:17.378119 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.378099    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:17.378221 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.378118    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:17.378221 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.378137    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:17.378221 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.378101    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.379064 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.379004    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:17.380358 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.380337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.380835 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.380817    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:17.381046 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.381023    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:17.381652 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.381631    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wt9jn\""
Sep 09 20:11:17.382675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.382650    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:17.382805 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.382729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dn758\""
Sep 09 20:11:17.383057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.383041    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.383395 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.383057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:17.383554 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.383378    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:17.384115 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d2b49f9-ccd9-486c-9328-04e63706a2a9-cni-binary-copy\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.384208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-daemon-config\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.384208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384147    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-multus-certs\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.384208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-host\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.384208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e732625a-d3ab-482c-a2c4-ad63e4786a0a-tmp\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-run-netns\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-socket-dir-parent\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-tuned\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdrp\" (UniqueName: \"kubernetes.io/projected/e732625a-d3ab-482c-a2c4-ad63e4786a0a-kube-api-access-hmdrp\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-log-socket\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:17.384421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-device-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-conf-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysctl-d\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-kubelet\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-systemd\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovnkube-config\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-var-lib-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-serviceca\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.384743 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-system-cni-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-etc-kubernetes\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-systemd-units\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-cnibin\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysctl-conf\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovn-node-metrics-cert\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a62488d-cc14-4591-a50f-1665c9ad2800-hosts-file\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.384984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-systemd\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-cni-bin\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-host\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.385204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qts\" (UniqueName: \"kubernetes.io/projected/38d7c169-a7a9-4978-88ba-1a166aa21947-kube-api-access-n8qts\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-node-log\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-env-overrides\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-cni-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-k8s-cni-cncf-io\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-kubelet\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-cni-netd\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vr5\" (UniqueName: \"kubernetes.io/projected/b82edf24-8b7c-460e-a586-3cced8f0ce73-kube-api-access-g7vr5\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-cni-bin\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-run\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-modprobe-d\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznkv\" (UniqueName: \"kubernetes.io/projected/9d2b49f9-ccd9-486c-9328-04e63706a2a9-kube-api-access-qznkv\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysconfig\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-sys\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.385783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-var-lib-kubelet\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-etc-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-run-ovn-kubernetes\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhw4\" (UniqueName: \"kubernetes.io/projected/fb165687-e762-406c-b185-f1ec0037f4cc-kube-api-access-mfhw4\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-netns\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-cni-multus\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-slash\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-ovn\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664b4\" (UniqueName: \"kubernetes.io/projected/5a62488d-cc14-4591-a50f-1665c9ad2800-kube-api-access-664b4\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-hostroot\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-kubernetes\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-lib-modules\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.385995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovnkube-script-lib\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.386031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a62488d-cc14-4591-a50f-1665c9ad2800-tmp-dir\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.386367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.386053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mm9h\" (UniqueName: \"kubernetes.io/projected/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-kube-api-access-9mm9h\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.387991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.387952    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:17.387991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.387952    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:17.388231 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.388207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2fwlt\""
Sep 09 20:11:17.388510 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.388491    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.394249 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.394221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:17.396428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.396392    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:17.396613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.396551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-chlqn\""
Sep 09 20:11:17.396714 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.396628    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:17.398728 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.398700    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:17.418554 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.418525    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wrgsh"
Sep 09 20:11:17.428677 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.428652    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wrgsh"
Sep 09 20:11:17.475681 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.475648    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:17.487154 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d2b49f9-ccd9-486c-9328-04e63706a2a9-cni-binary-copy\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.487353 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-daemon-config\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.487353 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-multus-certs\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.487353 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-host\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.487353 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e732625a-d3ab-482c-a2c4-ad63e4786a0a-tmp\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.487353 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-run-netns\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.487353 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-socket-dir-parent\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-tuned\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-host\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdrp\" (UniqueName: \"kubernetes.io/projected/e732625a-d3ab-482c-a2c4-ad63e4786a0a-kube-api-access-hmdrp\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-log-socket\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbzm\" (UniqueName: \"kubernetes.io/projected/d69639c4-1be8-4047-a7c4-8c29ad94358b-kube-api-access-qzbzm\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-multus-certs\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.487557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:17.487836 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.487836 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-socket-dir-parent\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.487897 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.487844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d2b49f9-ccd9-486c-9328-04e63706a2a9-cni-binary-copy\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-device-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488106    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-log-socket\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-conf-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-run-netns\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysctl-d\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-conf-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-kubelet\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-device-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-systemd\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488302    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysctl-d\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovnkube-config\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7365b6cd-5c52-4083-aeec-55909a783f40-konnectivity-ca\") pod \"konnectivity-agent-tvmd6\" (UID: \"7365b6cd-5c52-4083-aeec-55909a783f40\") " pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-systemd\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-kubelet\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-var-lib-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-serviceca\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.488563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-var-lib-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-iptables-alerter-script\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-system-cni-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-etc-kubernetes\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78e54e15-216a-4f30-ac4e-b1848d33090b-kubelet-config\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.488761    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-daemon-config\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-systemd-units\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.488844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-systemd-units\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.488908    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs podName:38d7c169-a7a9-4978-88ba-1a166aa21947 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:17.988849819 +0000 UTC m=+3.024320976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs") pod "network-metrics-daemon-qhxc4" (UID: "38d7c169-a7a9-4978-88ba-1a166aa21947") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovnkube-config\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-serviceca\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-system-cni-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-system-cni-dir\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.489558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489139    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-registration-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-cnibin\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-etc-kubernetes\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysctl-conf\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489284    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-cnibin\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovn-node-metrics-cert\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysctl-conf\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a62488d-cc14-4591-a50f-1665c9ad2800-hosts-file\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a62488d-cc14-4591-a50f-1665c9ad2800-hosts-file\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489573    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-host-slash\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-systemd\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489766    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-socket-dir\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.490760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-cni-bin\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-systemd\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-host\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qts\" (UniqueName: \"kubernetes.io/projected/38d7c169-a7a9-4978-88ba-1a166aa21947-kube-api-access-n8qts\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489851    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-cni-bin\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78e54e15-216a-4f30-ac4e-b1848d33090b-dbus\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-host\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.489969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb165687-e762-406c-b185-f1ec0037f4cc-sys-fs\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-node-log\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-env-overrides\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-node-log\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-cni-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-k8s-cni-cncf-io\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-kubelet\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.491782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-cni-netd\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vr5\" (UniqueName: \"kubernetes.io/projected/b82edf24-8b7c-460e-a586-3cced8f0ce73-kube-api-access-g7vr5\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-k8s-cni-cncf-io\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-multus-cni-dir\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-cni-netd\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7365b6cd-5c52-4083-aeec-55909a783f40-agent-certs\") pod \"konnectivity-agent-tvmd6\" (UID: \"7365b6cd-5c52-4083-aeec-55909a783f40\") " pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-cni-bin\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-run\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-kubelet\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-env-overrides\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-cnibin\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-cni-bin\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwzbj\" (UniqueName: \"kubernetes.io/projected/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-kube-api-access-lwzbj\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-run\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-modprobe-d\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.494379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznkv\" (UniqueName: \"kubernetes.io/projected/9d2b49f9-ccd9-486c-9328-04e63706a2a9-kube-api-access-qznkv\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysconfig\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-sys\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-var-lib-kubelet\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-etc-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-sysconfig\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-run-ovn-kubernetes\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhw4\" (UniqueName: \"kubernetes.io/projected/fb165687-e762-406c-b185-f1ec0037f4cc-kube-api-access-mfhw4\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-var-lib-kubelet\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-sys\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-netns\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-cni-multus\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-run-ovn-kubernetes\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-etc-openvswitch\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-modprobe-d\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-slash\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-run-netns\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-ovn\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.495312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664b4\" (UniqueName: \"kubernetes.io/projected/5a62488d-cc14-4591-a50f-1665c9ad2800-kube-api-access-664b4\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-host-slash\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b82edf24-8b7c-460e-a586-3cced8f0ce73-run-ovn\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.490985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-host-var-lib-cni-multus\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-hostroot\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-kubernetes\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-lib-modules\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9d2b49f9-ccd9-486c-9328-04e63706a2a9-hostroot\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovnkube-script-lib\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a62488d-cc14-4591-a50f-1665c9ad2800-tmp-dir\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491167    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-kubernetes\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e732625a-d3ab-482c-a2c4-ad63e4786a0a-lib-modules\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mm9h\" (UniqueName: \"kubernetes.io/projected/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-kube-api-access-9mm9h\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a62488d-cc14-4591-a50f-1665c9ad2800-tmp-dir\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovnkube-script-lib\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.491931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal" event={"ID":"7b0eae04d84f726f4b8e2b038df85374","Type":"ContainerStarted","Data":"28d3221bd988ccffb5c9459f1d5dd7c66be4ad1bcde7a90865f4ec43095e4542"}
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.493078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e732625a-d3ab-482c-a2c4-ad63e4786a0a-etc-tuned\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.493234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b82edf24-8b7c-460e-a586-3cced8f0ce73-ovn-node-metrics-cert\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.497208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.493248    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e732625a-d3ab-482c-a2c4-ad63e4786a0a-tmp\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.497208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.493622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal" event={"ID":"b89797f28a6a6731f98f222654fb2c7f","Type":"ContainerStarted","Data":"19b41b49ad70d1575152fe1e4ddb5c577ad795f48ce5a59f19f456958157eaed"}
Sep 09 20:11:17.501019 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.500990    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:17.501019 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.501020    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:17.501219 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.501039    2566 projected.go:196] Error preparing data for projected volume kube-api-access-pmr2m for pod openshift-network-diagnostics/network-check-target-2vsjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:17.501219 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.501123    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m podName:ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef nodeName:}" failed. No retries permitted until 2026-09-09 20:11:18.001085337 +0000 UTC m=+3.036556495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pmr2m" (UniqueName: "kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m") pod "network-check-target-2vsjc" (UID: "ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:17.504289 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.504263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vr5\" (UniqueName: \"kubernetes.io/projected/b82edf24-8b7c-460e-a586-3cced8f0ce73-kube-api-access-g7vr5\") pod \"ovnkube-node-h7z6h\" (UID: \"b82edf24-8b7c-460e-a586-3cced8f0ce73\") " pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.504375 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.504289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664b4\" (UniqueName: \"kubernetes.io/projected/5a62488d-cc14-4591-a50f-1665c9ad2800-kube-api-access-664b4\") pod \"node-resolver-pt8r9\" (UID: \"5a62488d-cc14-4591-a50f-1665c9ad2800\") " pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.505192 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.505000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdrp\" (UniqueName: \"kubernetes.io/projected/e732625a-d3ab-482c-a2c4-ad63e4786a0a-kube-api-access-hmdrp\") pod \"tuned-tmptz\" (UID: \"e732625a-d3ab-482c-a2c4-ad63e4786a0a\") " pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.505333 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.505316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhw4\" (UniqueName: \"kubernetes.io/projected/fb165687-e762-406c-b185-f1ec0037f4cc-kube-api-access-mfhw4\") pod \"aws-ebs-csi-driver-node-k9mnf\" (UID: \"fb165687-e762-406c-b185-f1ec0037f4cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.505659 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.505641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mm9h\" (UniqueName: \"kubernetes.io/projected/b3ab512a-1904-4284-82da-8a9a0a2aa4e1-kube-api-access-9mm9h\") pod \"node-ca-rzvkf\" (UID: \"b3ab512a-1904-4284-82da-8a9a0a2aa4e1\") " pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.508185 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.507627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznkv\" (UniqueName: \"kubernetes.io/projected/9d2b49f9-ccd9-486c-9328-04e63706a2a9-kube-api-access-qznkv\") pod \"multus-fk4th\" (UID: \"9d2b49f9-ccd9-486c-9328-04e63706a2a9\") " pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.510076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.510055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qts\" (UniqueName: \"kubernetes.io/projected/38d7c169-a7a9-4978-88ba-1a166aa21947-kube-api-access-n8qts\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:17.515053 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.515034    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:17.592307 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-system-cni-dir\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.592307 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-host-slash\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78e54e15-216a-4f30-ac4e-b1848d33090b-dbus\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592400    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-system-cni-dir\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7365b6cd-5c52-4083-aeec-55909a783f40-agent-certs\") pod \"konnectivity-agent-tvmd6\" (UID: \"7365b6cd-5c52-4083-aeec-55909a783f40\") " pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-host-slash\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-cnibin\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwzbj\" (UniqueName: \"kubernetes.io/projected/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-kube-api-access-lwzbj\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.592578 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.592570    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78e54e15-216a-4f30-ac4e-b1848d33090b-dbus\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbzm\" (UniqueName: \"kubernetes.io/projected/d69639c4-1be8-4047-a7c4-8c29ad94358b-kube-api-access-qzbzm\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-cnibin\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.592651    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret podName:78e54e15-216a-4f30-ac4e-b1848d33090b nodeName:}" failed. No retries permitted until 2026-09-09 20:11:18.092629972 +0000 UTC m=+3.128101132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret") pod "global-pull-secret-syncer-tq7t4" (UID: "78e54e15-216a-4f30-ac4e-b1848d33090b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7365b6cd-5c52-4083-aeec-55909a783f40-konnectivity-ca\") pod \"konnectivity-agent-tvmd6\" (UID: \"7365b6cd-5c52-4083-aeec-55909a783f40\") " pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d69639c4-1be8-4047-a7c4-8c29ad94358b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-iptables-alerter-script\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.592976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.593003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.593007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78e54e15-216a-4f30-ac4e-b1848d33090b-kubelet-config\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.593076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.593049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78e54e15-216a-4f30-ac4e-b1848d33090b-kubelet-config\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:17.593659 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.593175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-cni-binary-copy\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.593659 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.593486    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-iptables-alerter-script\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.593659 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.593494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d69639c4-1be8-4047-a7c4-8c29ad94358b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.594379 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.594356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7365b6cd-5c52-4083-aeec-55909a783f40-konnectivity-ca\") pod \"konnectivity-agent-tvmd6\" (UID: \"7365b6cd-5c52-4083-aeec-55909a783f40\") " pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.595515 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.595495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7365b6cd-5c52-4083-aeec-55909a783f40-agent-certs\") pod \"konnectivity-agent-tvmd6\" (UID: \"7365b6cd-5c52-4083-aeec-55909a783f40\") " pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.603246 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.603188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbzm\" (UniqueName: \"kubernetes.io/projected/d69639c4-1be8-4047-a7c4-8c29ad94358b-kube-api-access-qzbzm\") pod \"multus-additional-cni-plugins-ddscx\" (UID: \"d69639c4-1be8-4047-a7c4-8c29ad94358b\") " pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.603921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.603901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwzbj\" (UniqueName: \"kubernetes.io/projected/a8d75e19-6ca5-4d7b-b04f-8ac162e844ea-kube-api-access-lwzbj\") pod \"iptables-alerter-w4lv4\" (UID: \"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea\") " pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.652456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.652412    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:17.676843 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.676806    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf"
Sep 09 20:11:17.687047 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.687023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fk4th"
Sep 09 20:11:17.698323 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.698294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tmptz"
Sep 09 20:11:17.704322 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.704300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzvkf"
Sep 09 20:11:17.711316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.711295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:17.720330 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.720306    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pt8r9"
Sep 09 20:11:17.726363 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.726331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ddscx"
Sep 09 20:11:17.736331 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.736300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:17.741349 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.741329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4lv4"
Sep 09 20:11:17.996753 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:17.996660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:17.996921 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.996785    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:17.996921 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:17.996853    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs podName:38d7c169-a7a9-4978-88ba-1a166aa21947 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:18.996832608 +0000 UTC m=+4.032303749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs") pod "network-metrics-daemon-qhxc4" (UID: "38d7c169-a7a9-4978-88ba-1a166aa21947") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:18.097993 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:18.097950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:18.098209 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:18.098047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:18.098209 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:18.098175    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:18.098209 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:18.098199    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:18.098209 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:18.098207    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:18.098426 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:18.098219    2566 projected.go:196] Error preparing data for projected volume kube-api-access-pmr2m for pod openshift-network-diagnostics/network-check-target-2vsjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:18.098426 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:18.098272    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret podName:78e54e15-216a-4f30-ac4e-b1848d33090b nodeName:}" failed. No retries permitted until 2026-09-09 20:11:19.098250701 +0000 UTC m=+4.133721847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret") pod "global-pull-secret-syncer-tq7t4" (UID: "78e54e15-216a-4f30-ac4e-b1848d33090b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:18.098426 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:18.098294    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m podName:ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef nodeName:}" failed. No retries permitted until 2026-09-09 20:11:19.098282921 +0000 UTC m=+4.133754064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmr2m" (UniqueName: "kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m") pod "network-check-target-2vsjc" (UID: "ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:18.432328 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:18.432227    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:17 +0000 UTC" deadline="2028-02-04 12:12:32.16386634 +0000 UTC"
Sep 09 20:11:18.432328 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:18.432267    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12304h1m13.731604231s"
Sep 09 20:11:18.540377 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:18.540353    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ab512a_1904_4284_82da_8a9a0a2aa4e1.slice/crio-6eb77e8b053e481ecc677eeaef1d79a90f4532e8190e8cceb1fbb2946e435bc5 WatchSource:0}: Error finding container 6eb77e8b053e481ecc677eeaef1d79a90f4532e8190e8cceb1fbb2946e435bc5: Status 404 returned error can't find the container with id 6eb77e8b053e481ecc677eeaef1d79a90f4532e8190e8cceb1fbb2946e435bc5
Sep 09 20:11:18.542219 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:18.542190    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7365b6cd_5c52_4083_aeec_55909a783f40.slice/crio-3e76b6830536f057a3a3476dab3a842723ea1419db42eabb7f1bab24e8b83acd WatchSource:0}: Error finding container 3e76b6830536f057a3a3476dab3a842723ea1419db42eabb7f1bab24e8b83acd: Status 404 returned error can't find the container with id 3e76b6830536f057a3a3476dab3a842723ea1419db42eabb7f1bab24e8b83acd
Sep 09 20:11:18.543298 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:18.543272    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69639c4_1be8_4047_a7c4_8c29ad94358b.slice/crio-0d59da7ab899d7f599a805eb7b194284cb2c5a3c47a1894153edd93422425060 WatchSource:0}: Error finding container 0d59da7ab899d7f599a805eb7b194284cb2c5a3c47a1894153edd93422425060: Status 404 returned error can't find the container with id 0d59da7ab899d7f599a805eb7b194284cb2c5a3c47a1894153edd93422425060
Sep 09 20:11:18.544222 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:18.544071    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a62488d_cc14_4591_a50f_1665c9ad2800.slice/crio-0e40a1aaf8641a3781a996c4aa62676a7bd71e1449ed1a11afd143b883692f2e WatchSource:0}: Error finding container 0e40a1aaf8641a3781a996c4aa62676a7bd71e1449ed1a11afd143b883692f2e: Status 404 returned error can't find the container with id 0e40a1aaf8641a3781a996c4aa62676a7bd71e1449ed1a11afd143b883692f2e
Sep 09 20:11:18.568767 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:18.568725    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode732625a_d3ab_482c_a2c4_ad63e4786a0a.slice/crio-3a8cfdbc63a9fccaa293265ed8f6a66af2cc084f8af2b1b131661f4b1a853222 WatchSource:0}: Error finding container 3a8cfdbc63a9fccaa293265ed8f6a66af2cc084f8af2b1b131661f4b1a853222: Status 404 returned error can't find the container with id 3a8cfdbc63a9fccaa293265ed8f6a66af2cc084f8af2b1b131661f4b1a853222
Sep 09 20:11:18.569898 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:18.569858    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb165687_e762_406c_b185_f1ec0037f4cc.slice/crio-62b10af179e738bc9b4ae1928428cc9dee515ce0d0339e2338f7e463bd0eeced WatchSource:0}: Error finding container 62b10af179e738bc9b4ae1928428cc9dee515ce0d0339e2338f7e463bd0eeced: Status 404 returned error can't find the container with id 62b10af179e738bc9b4ae1928428cc9dee515ce0d0339e2338f7e463bd0eeced
Sep 09 20:11:19.014783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.014207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:19.014783 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.014365    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:19.014783 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.014435    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs podName:38d7c169-a7a9-4978-88ba-1a166aa21947 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:21.014414416 +0000 UTC m=+6.049885559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs") pod "network-metrics-daemon-qhxc4" (UID: "38d7c169-a7a9-4978-88ba-1a166aa21947") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:19.114978 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.114886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:19.115134 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.114991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:19.115236 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.115153    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:19.115236 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.115222    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret podName:78e54e15-216a-4f30-ac4e-b1848d33090b nodeName:}" failed. No retries permitted until 2026-09-09 20:11:21.115202181 +0000 UTC m=+6.150673328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret") pod "global-pull-secret-syncer-tq7t4" (UID: "78e54e15-216a-4f30-ac4e-b1848d33090b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:19.115712 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.115684    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:19.115796 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.115719    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:19.115796 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.115734    2566 projected.go:196] Error preparing data for projected volume kube-api-access-pmr2m for pod openshift-network-diagnostics/network-check-target-2vsjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:19.115796 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.115781    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m podName:ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef nodeName:}" failed. No retries permitted until 2026-09-09 20:11:21.115764608 +0000 UTC m=+6.151235753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmr2m" (UniqueName: "kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m") pod "network-check-target-2vsjc" (UID: "ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:19.433503 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.433378    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:17 +0000 UTC" deadline="2028-02-23 22:49:01.537444385 +0000 UTC"
Sep 09 20:11:19.433503 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.433446    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12770h37m42.104002362s"
Sep 09 20:11:19.487113 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.486286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:19.487113 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.486423    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:19.487113 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.486879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:19.487113 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.486994    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:19.487823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.487634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:19.487823 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:19.487744    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:19.510802 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.510767    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"9e913c5a3955c62c03c623632bbe2d223e2018f975125dbdfc15c3360cca4f8e"}
Sep 09 20:11:19.515250 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.515194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pt8r9" event={"ID":"5a62488d-cc14-4591-a50f-1665c9ad2800","Type":"ContainerStarted","Data":"0e40a1aaf8641a3781a996c4aa62676a7bd71e1449ed1a11afd143b883692f2e"}
Sep 09 20:11:19.518257 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.517984    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerStarted","Data":"0d59da7ab899d7f599a805eb7b194284cb2c5a3c47a1894153edd93422425060"}
Sep 09 20:11:19.528518 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.528461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tvmd6" event={"ID":"7365b6cd-5c52-4083-aeec-55909a783f40","Type":"ContainerStarted","Data":"3e76b6830536f057a3a3476dab3a842723ea1419db42eabb7f1bab24e8b83acd"}
Sep 09 20:11:19.535380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.535294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzvkf" event={"ID":"b3ab512a-1904-4284-82da-8a9a0a2aa4e1","Type":"ContainerStarted","Data":"6eb77e8b053e481ecc677eeaef1d79a90f4532e8190e8cceb1fbb2946e435bc5"}
Sep 09 20:11:19.542898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.542217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal" event={"ID":"7b0eae04d84f726f4b8e2b038df85374","Type":"ContainerStarted","Data":"11d0b73d52f5aa7c323f535d5d5986014f4e764e67cca0c50870010ad69002bf"}
Sep 09 20:11:19.554201 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.554095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4lv4" event={"ID":"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea","Type":"ContainerStarted","Data":"763a154742b060222bcedd0c82cb84fe962d99cd3201a0a478c6facfb6181fa2"}
Sep 09 20:11:19.558903 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.558838    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-185.ec2.internal" podStartSLOduration=3.558820526 podStartE2EDuration="3.558820526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:19.558534355 +0000 UTC m=+4.594005524" watchObservedRunningTime="2026-09-09 20:11:19.558820526 +0000 UTC m=+4.594291691"
Sep 09 20:11:19.560157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.560119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf" event={"ID":"fb165687-e762-406c-b185-f1ec0037f4cc","Type":"ContainerStarted","Data":"62b10af179e738bc9b4ae1928428cc9dee515ce0d0339e2338f7e463bd0eeced"}
Sep 09 20:11:19.567147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.567111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tmptz" event={"ID":"e732625a-d3ab-482c-a2c4-ad63e4786a0a","Type":"ContainerStarted","Data":"3a8cfdbc63a9fccaa293265ed8f6a66af2cc084f8af2b1b131661f4b1a853222"}
Sep 09 20:11:19.573640 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.573603    2566 generic.go:356] "Generic (PLEG): container finished" podID="b89797f28a6a6731f98f222654fb2c7f" containerID="adcf9b57e31f887188fb5bf982095693c2dcc5d49d5dde4738e292fc6174dcc2" exitCode=0
Sep 09 20:11:19.573989 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.573918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal" event={"ID":"b89797f28a6a6731f98f222654fb2c7f","Type":"ContainerDied","Data":"adcf9b57e31f887188fb5bf982095693c2dcc5d49d5dde4738e292fc6174dcc2"}
Sep 09 20:11:19.576705 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:19.576674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fk4th" event={"ID":"9d2b49f9-ccd9-486c-9328-04e63706a2a9","Type":"ContainerStarted","Data":"8baf05d2f14ee5fdd2f044192a2a86461d5a7da09a836ae040f6379529e73fe7"}
Sep 09 20:11:20.598263 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:20.598224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal" event={"ID":"b89797f28a6a6731f98f222654fb2c7f","Type":"ContainerStarted","Data":"cdc7e142df88701be745444ffaabbe682464b190e29cc830421245ed12be7f90"}
Sep 09 20:11:20.615431 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:20.615357    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-185.ec2.internal" podStartSLOduration=4.615334447 podStartE2EDuration="4.615334447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:20.614377549 +0000 UTC m=+5.649848706" watchObservedRunningTime="2026-09-09 20:11:20.615334447 +0000 UTC m=+5.650805610"
Sep 09 20:11:21.033005 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:21.032367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:21.033005 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.032556    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:21.033005 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.032620    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs podName:38d7c169-a7a9-4978-88ba-1a166aa21947 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:25.032602602 +0000 UTC m=+10.068073754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs") pod "network-metrics-daemon-qhxc4" (UID: "38d7c169-a7a9-4978-88ba-1a166aa21947") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:21.133139 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:21.133104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:21.133302 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:21.133171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:21.133378 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.133319    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:21.133378 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.133340    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:21.133378 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.133352    2566 projected.go:196] Error preparing data for projected volume kube-api-access-pmr2m for pod openshift-network-diagnostics/network-check-target-2vsjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:21.133522 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.133418    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m podName:ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef nodeName:}" failed. No retries permitted until 2026-09-09 20:11:25.133390455 +0000 UTC m=+10.168861612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmr2m" (UniqueName: "kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m") pod "network-check-target-2vsjc" (UID: "ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:21.133839 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.133817    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:21.133938 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.133871    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret podName:78e54e15-216a-4f30-ac4e-b1848d33090b nodeName:}" failed. No retries permitted until 2026-09-09 20:11:25.13385753 +0000 UTC m=+10.169328671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret") pod "global-pull-secret-syncer-tq7t4" (UID: "78e54e15-216a-4f30-ac4e-b1848d33090b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:21.486931 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:21.486207    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:21.486931 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.486346    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:21.486931 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:21.486717    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:21.486931 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.486804    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:21.487447 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:21.487288    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:21.487447 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:21.487404    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:23.489197 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:23.488397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:23.489197 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:23.488529    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:23.489197 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:23.488603    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:23.489197 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:23.488682    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:23.489197 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:23.488730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:23.489197 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:23.488805    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:25.068660 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:25.068625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:25.069066 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.068749    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:25.069066 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.068809    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs podName:38d7c169-a7a9-4978-88ba-1a166aa21947 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:33.068792906 +0000 UTC m=+18.104264043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs") pod "network-metrics-daemon-qhxc4" (UID: "38d7c169-a7a9-4978-88ba-1a166aa21947") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:25.169618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:25.169578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:25.169785 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:25.169655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:25.169785 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.169757    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:25.169906 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.169781    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:25.169906 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.169807    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret podName:78e54e15-216a-4f30-ac4e-b1848d33090b nodeName:}" failed. No retries permitted until 2026-09-09 20:11:33.169793434 +0000 UTC m=+18.205264572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret") pod "global-pull-secret-syncer-tq7t4" (UID: "78e54e15-216a-4f30-ac4e-b1848d33090b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:25.169906 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.169807    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:25.169906 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.169822    2566 projected.go:196] Error preparing data for projected volume kube-api-access-pmr2m for pod openshift-network-diagnostics/network-check-target-2vsjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:25.169906 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.169860    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m podName:ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef nodeName:}" failed. No retries permitted until 2026-09-09 20:11:33.169848196 +0000 UTC m=+18.205319334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmr2m" (UniqueName: "kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m") pod "network-check-target-2vsjc" (UID: "ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:25.487006 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:25.486706    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:25.487155 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.487050    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:25.487411 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:25.486735    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:25.487478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:25.487419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:25.487531 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.487515    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:25.487649 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:25.487594    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:27.487645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:27.487198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:27.487645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:27.487216    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:27.487645 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:27.487309    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:27.487645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:27.487359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:27.487645 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:27.487446    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:27.487645 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:27.487544    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:29.485407 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:29.485343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:29.485848 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:29.485356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:29.485848 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:29.485502    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:29.485848 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:29.485510    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:29.485848 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:29.485577    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:29.485848 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:29.485639    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:31.485567 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:31.485525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:31.486015 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:31.485661    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:31.486015 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:31.485531    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:31.486015 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:31.485690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:31.486015 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:31.485760    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:31.486015 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:31.485844    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:33.131392 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:33.131350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:33.131809 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.131493    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:33.131809 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.131552    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs podName:38d7c169-a7a9-4978-88ba-1a166aa21947 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:49.131536604 +0000 UTC m=+34.167007743 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs") pod "network-metrics-daemon-qhxc4" (UID: "38d7c169-a7a9-4978-88ba-1a166aa21947") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:33.232666 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:33.232630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:33.232813 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:33.232697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:33.232813 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.232791    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:33.232813 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.232793    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:33.232813 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.232810    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:33.232929 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.232821    2566 projected.go:196] Error preparing data for projected volume kube-api-access-pmr2m for pod openshift-network-diagnostics/network-check-target-2vsjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:33.232929 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.232866    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret podName:78e54e15-216a-4f30-ac4e-b1848d33090b nodeName:}" failed. No retries permitted until 2026-09-09 20:11:49.232849314 +0000 UTC m=+34.268320452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret") pod "global-pull-secret-syncer-tq7t4" (UID: "78e54e15-216a-4f30-ac4e-b1848d33090b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:33.232929 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.232885    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m podName:ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef nodeName:}" failed. No retries permitted until 2026-09-09 20:11:49.232877057 +0000 UTC m=+34.268348200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmr2m" (UniqueName: "kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m") pod "network-check-target-2vsjc" (UID: "ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:33.485136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:33.485100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:33.485136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:33.485140    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:33.485312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:33.485112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:33.485312 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.485233    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:33.485312 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.485296    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:33.485417 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:33.485394    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:35.486401 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:35.486363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:35.486876 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:35.486485    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:35.486876 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:35.486505    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:35.486876 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:35.486596    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:35.486876 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:35.486622    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:35.486876 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:35.486691    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:37.486267 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.485964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:37.487075 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.485954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:37.487075 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:37.486371    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:37.487075 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:37.486417    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:37.487075 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.486028    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:37.487075 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:37.486534    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:37.642569 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.642484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fk4th" event={"ID":"9d2b49f9-ccd9-486c-9328-04e63706a2a9","Type":"ContainerStarted","Data":"2a54e01e6205aead917a1d0ad8497ea1d55b4a962f0dba17a8a297287c5251e1"}
Sep 09 20:11:37.645557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.645530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"926a0c7084470573112c26ae93754eacc62673c77c4f5709299c9b43fbbe168a"}
Sep 09 20:11:37.645693 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.645609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"7762f82fcd6fc179cc45fb3f0e9911b782077baacb957e861027b1460826d994"}
Sep 09 20:11:37.645693 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.645659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"2d2a0e48e20486da790bf1a302f24a711ec054f016a0dbba4994f416121f9d30"}
Sep 09 20:11:37.645773 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.645710    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"f1faf37df10a46dbdce72a76dbc9601b21e244fb65b8e34d04fbbd24d9342920"}
Sep 09 20:11:37.645773 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.645756    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"fdb472551dd275dd18c429349683455ef75691b82ba0ce0551cca0dbc3600bdc"}
Sep 09 20:11:37.645846 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.645787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"9d15781337275794f544cdad1b34c15318170d150dd8abf1c4f742ba69d79cbe"}
Sep 09 20:11:37.647098 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.647056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pt8r9" event={"ID":"5a62488d-cc14-4591-a50f-1665c9ad2800","Type":"ContainerStarted","Data":"0ac78ddceaba34b65b472d7716f2d30a2b874bf083ba8c85c9683bef9fbc9486"}
Sep 09 20:11:37.648640 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.648621    2566 generic.go:356] "Generic (PLEG): container finished" podID="d69639c4-1be8-4047-a7c4-8c29ad94358b" containerID="c4133f18bdf3b3a0f6c5b695cbcc532a581d587e3602cff6e2409048f8ec17db" exitCode=0
Sep 09 20:11:37.648742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.648680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerDied","Data":"c4133f18bdf3b3a0f6c5b695cbcc532a581d587e3602cff6e2409048f8ec17db"}
Sep 09 20:11:37.650228 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.650201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tvmd6" event={"ID":"7365b6cd-5c52-4083-aeec-55909a783f40","Type":"ContainerStarted","Data":"04a8b201b966cfe70e25a80725d6e6e4bf5e196004b3fe3e17e0fd85d67c9245"}
Sep 09 20:11:37.651848 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.651824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzvkf" event={"ID":"b3ab512a-1904-4284-82da-8a9a0a2aa4e1","Type":"ContainerStarted","Data":"f219a7324dd8bc766f000c487f6f8f0da61846de69ea14f70f4ee58879ae3f47"}
Sep 09 20:11:37.653204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.653185    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf" event={"ID":"fb165687-e762-406c-b185-f1ec0037f4cc","Type":"ContainerStarted","Data":"504abffb7882e5a56883c3c7f1ba0072ea16b1ad9ab79ccba73d1fc77bc9e981"}
Sep 09 20:11:37.654600 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.654581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tmptz" event={"ID":"e732625a-d3ab-482c-a2c4-ad63e4786a0a","Type":"ContainerStarted","Data":"0e6b4aba070ca2c12ae9bd2410e2251c86cdb92244674ac55b8f6590663c423e"}
Sep 09 20:11:37.659235 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.659200    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fk4th" podStartSLOduration=4.353392066 podStartE2EDuration="22.659189044s" totalImagesPullingTime="18.305796978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:37.658732764 +0000 UTC m=+22.694203923" watchObservedRunningTime="2026-09-09 20:11:37.659189044 +0000 UTC m=+22.694660204"
Sep 09 20:11:37.673463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.673413    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rzvkf" podStartSLOduration=4.380561131 podStartE2EDuration="22.673377312s" totalImagesPullingTime="18.292816181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:37.673106239 +0000 UTC m=+22.708577392" watchObservedRunningTime="2026-09-09 20:11:37.673377312 +0000 UTC m=+22.708848473"
Sep 09 20:11:37.687796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.687748    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pt8r9" podStartSLOduration=4.402508627 podStartE2EDuration="22.687732856s" totalImagesPullingTime="18.285224229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:37.687527165 +0000 UTC m=+22.722998324" watchObservedRunningTime="2026-09-09 20:11:37.687732856 +0000 UTC m=+22.723204015"
Sep 09 20:11:37.726835 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.726785    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tmptz" podStartSLOduration=4.420491177 podStartE2EDuration="22.726767722s" totalImagesPullingTime="18.306276545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:37.72671119 +0000 UTC m=+22.762182355" watchObservedRunningTime="2026-09-09 20:11:37.726767722 +0000 UTC m=+22.762238912"
Sep 09 20:11:37.727214 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:37.727186    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tvmd6" podStartSLOduration=4.436574857 podStartE2EDuration="22.72717577s" totalImagesPullingTime="18.290600913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:37.703785763 +0000 UTC m=+22.739256921" watchObservedRunningTime="2026-09-09 20:11:37.72717577 +0000 UTC m=+22.762646929"
Sep 09 20:11:38.366027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:38.365862    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:38.463606 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:38.463379    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:38.366023276Z","UUID":"0ef8a3b0-953f-4059-97c3-347537e0f973","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:38.465169 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:38.465149    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:38.465301 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:38.465182    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:38.658345 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:38.658311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4lv4" event={"ID":"a8d75e19-6ca5-4d7b-b04f-8ac162e844ea","Type":"ContainerStarted","Data":"74287c0a5fed8d880d1004ffb8414fd492d954c605e48ee65a64c6f38c554882"}
Sep 09 20:11:38.660256 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:38.660221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf" event={"ID":"fb165687-e762-406c-b185-f1ec0037f4cc","Type":"ContainerStarted","Data":"8148727e3050990fb70e32d855d65b7c27413ee39673a21aefe386c258039b1f"}
Sep 09 20:11:38.674190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:38.674123    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w4lv4" podStartSLOduration=5.408797247 podStartE2EDuration="23.674099634s" totalImagesPullingTime="18.265302387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:38.673719715 +0000 UTC m=+23.709190869" watchObservedRunningTime="2026-09-09 20:11:38.674099634 +0000 UTC m=+23.709570789"
Sep 09 20:11:39.485579 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.485553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:39.485715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.485562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:39.485715 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:39.485676    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:39.485827 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:39.485808    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:39.485887 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.485877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:39.485979 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:39.485961    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:39.665443 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.665321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf" event={"ID":"fb165687-e762-406c-b185-f1ec0037f4cc","Type":"ContainerStarted","Data":"5e07e817383fc3d875e2e6bab97a93a0b2e9b75d26f210b8d471b7dca2d72de3"}
Sep 09 20:11:39.670005 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.669848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"b1b676115caf6d1a7698b2e6f9f9da8f8b27aa2037d0993c23c37d79d89eeae2"}
Sep 09 20:11:39.684740 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.684689    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k9mnf" podStartSLOduration=4.288657862 podStartE2EDuration="24.684669575s" totalImagesPullingTime="20.396011713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:39.684573651 +0000 UTC m=+24.720044809" watchObservedRunningTime="2026-09-09 20:11:39.684669575 +0000 UTC m=+24.720140740"
Sep 09 20:11:39.716906 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.716866    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:39.717934 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:39.717912    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:40.672841 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:40.672808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:40.673329 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:40.673313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tvmd6"
Sep 09 20:11:41.485162 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:41.485125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:41.485332 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:41.485180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:41.485332 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:41.485254    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:41.485332 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:41.485130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:41.485547 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:41.485399    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:41.485547 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:41.485509    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:42.678125 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:42.677919    2566 generic.go:356] "Generic (PLEG): container finished" podID="d69639c4-1be8-4047-a7c4-8c29ad94358b" containerID="bb670a33e10e34bd3fbbdfa574992be330bd3ee01d5e4e8ac66b480d17237a91" exitCode=0
Sep 09 20:11:42.678677 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:42.678007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerDied","Data":"bb670a33e10e34bd3fbbdfa574992be330bd3ee01d5e4e8ac66b480d17237a91"}
Sep 09 20:11:42.682411 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:42.682374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" event={"ID":"b82edf24-8b7c-460e-a586-3cced8f0ce73","Type":"ContainerStarted","Data":"090f92b59a07b900df9bd551a2b4869b0cb4549e6a31151e8c6425aad5c53c53"}
Sep 09 20:11:43.485563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.485528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:43.485802 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.485528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:43.485802 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:43.485638    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:43.485802 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.485528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:43.485802 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:43.485733    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:43.485943 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:43.485823    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:43.685147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.685118    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:43.685903 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.685213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:43.685903 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.685262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:43.703728 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.703693    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:43.703898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.703845    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:11:43.734653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.734593    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h" podStartSLOduration=10.363920098 podStartE2EDuration="28.734576174s" totalImagesPullingTime="18.370656076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:42.732193033 +0000 UTC m=+27.767664191" watchObservedRunningTime="2026-09-09 20:11:43.734576174 +0000 UTC m=+28.770047333"
Sep 09 20:11:43.813709 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.813687    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tq7t4"]
Sep 09 20:11:43.813825 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.813787    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:43.813893 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:43.813873    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:43.816604 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.816580    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vsjc"]
Sep 09 20:11:43.816756 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.816743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:43.816855 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:43.816836    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:43.817130 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.817114    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhxc4"]
Sep 09 20:11:43.817191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:43.817181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:43.817268 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:43.817253    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:44.689770 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:44.689567    2566 generic.go:356] "Generic (PLEG): container finished" podID="d69639c4-1be8-4047-a7c4-8c29ad94358b" containerID="9bc660f83d39493ec4bf475d048115439b2da4cde8a0965494bd16f4fca6ebfa" exitCode=0
Sep 09 20:11:44.689770 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:44.689647    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerDied","Data":"9bc660f83d39493ec4bf475d048115439b2da4cde8a0965494bd16f4fca6ebfa"}
Sep 09 20:11:45.486560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:45.486531    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:45.486745 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:45.486621    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:45.486745 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:45.486634    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:45.486745 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:45.486699    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:45.486745 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:45.486708    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:45.486939 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:45.486756    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:46.696566 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:46.696533    2566 generic.go:356] "Generic (PLEG): container finished" podID="d69639c4-1be8-4047-a7c4-8c29ad94358b" containerID="a1c3d40e444f9f11fdc9b465da651a4fb695907ca7ee0819dd5fbc529a01b931" exitCode=0
Sep 09 20:11:46.696914 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:46.696584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerDied","Data":"a1c3d40e444f9f11fdc9b465da651a4fb695907ca7ee0819dd5fbc529a01b931"}
Sep 09 20:11:47.485759 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:47.485719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:47.485954 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:47.485719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:47.485954 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:47.485846    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:47.485954 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:47.485719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:47.485954 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:47.485927    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:47.486161 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:47.486002    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:49.142808 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:49.142779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:49.143351 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.142903    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:49.143351 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.142961    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs podName:38d7c169-a7a9-4978-88ba-1a166aa21947 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:21.142945843 +0000 UTC m=+66.178416989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs") pod "network-metrics-daemon-qhxc4" (UID: "38d7c169-a7a9-4978-88ba-1a166aa21947") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:49.243297 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:49.243267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:49.243468 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:49.243321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:49.243468 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.243406    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:49.243468 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.243437    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:49.243468 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.243457    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret podName:78e54e15-216a-4f30-ac4e-b1848d33090b nodeName:}" failed. No retries permitted until 2026-09-09 20:12:21.243444341 +0000 UTC m=+66.278915478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret") pod "global-pull-secret-syncer-tq7t4" (UID: "78e54e15-216a-4f30-ac4e-b1848d33090b") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:49.243468 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.243460    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:49.243660 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.243474    2566 projected.go:196] Error preparing data for projected volume kube-api-access-pmr2m for pod openshift-network-diagnostics/network-check-target-2vsjc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:49.243660 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.243525    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m podName:ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef nodeName:}" failed. No retries permitted until 2026-09-09 20:12:21.243506923 +0000 UTC m=+66.278978060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pmr2m" (UniqueName: "kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m") pod "network-check-target-2vsjc" (UID: "ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:49.485495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:49.485354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:49.485495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:49.485391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:49.485680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:49.485495    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:49.485680 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.485509    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vsjc" podUID="ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef"
Sep 09 20:11:49.485680 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.485589    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qhxc4" podUID="38d7c169-a7a9-4978-88ba-1a166aa21947"
Sep 09 20:11:49.485680 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:49.485642    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tq7t4" podUID="78e54e15-216a-4f30-ac4e-b1848d33090b"
Sep 09 20:11:50.709927 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.709894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerStarted","Data":"765979619f0cd151e86465fd1e7b32b3d2dbf121617218e27c9c05de95c3732c"}
Sep 09 20:11:50.770525 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.770488    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-185.ec2.internal" event="NodeReady"
Sep 09 20:11:50.770679 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.770663    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:11:50.818601 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.818567    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jthx2"]
Sep 09 20:11:50.821347 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.821318    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qdgdx"]
Sep 09 20:11:50.821469 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.821448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:50.823826 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.823804    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:11:50.823936 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.823833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:11:50.823936 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.823869    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftsch\""
Sep 09 20:11:50.824040 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.823950    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:50.826271 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.826255    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:11:50.826357 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.826305    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:11:50.826396 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.826373    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7rf7m\""
Sep 09 20:11:50.826462 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.826449    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:11:50.829881 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.829864    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qdgdx"]
Sep 09 20:11:50.832995 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.832941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jthx2"]
Sep 09 20:11:50.957152 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.957053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rg4\" (UniqueName: \"kubernetes.io/projected/1e40077b-94d8-4b82-8982-969b50b1fe96-kube-api-access-v8rg4\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:50.957152 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.957107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5739f528-c67f-4c93-807e-80854f5b8b06-cert\") pod \"ingress-canary-qdgdx\" (UID: \"5739f528-c67f-4c93-807e-80854f5b8b06\") " pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:50.957331 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.957161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mf6\" (UniqueName: \"kubernetes.io/projected/5739f528-c67f-4c93-807e-80854f5b8b06-kube-api-access-g2mf6\") pod \"ingress-canary-qdgdx\" (UID: \"5739f528-c67f-4c93-807e-80854f5b8b06\") " pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:50.957331 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.957197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:50.957331 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.957244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e40077b-94d8-4b82-8982-969b50b1fe96-tmp-dir\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:50.957331 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:50.957285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e40077b-94d8-4b82-8982-969b50b1fe96-config-volume\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.058355 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.058321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rg4\" (UniqueName: \"kubernetes.io/projected/1e40077b-94d8-4b82-8982-969b50b1fe96-kube-api-access-v8rg4\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.058355 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.058356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5739f528-c67f-4c93-807e-80854f5b8b06-cert\") pod \"ingress-canary-qdgdx\" (UID: \"5739f528-c67f-4c93-807e-80854f5b8b06\") " pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:51.058589 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.058381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mf6\" (UniqueName: \"kubernetes.io/projected/5739f528-c67f-4c93-807e-80854f5b8b06-kube-api-access-g2mf6\") pod \"ingress-canary-qdgdx\" (UID: \"5739f528-c67f-4c93-807e-80854f5b8b06\") " pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:51.058589 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.058409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.058589 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.058436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e40077b-94d8-4b82-8982-969b50b1fe96-tmp-dir\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.058589 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.058489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e40077b-94d8-4b82-8982-969b50b1fe96-config-volume\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.058589 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:51.058530    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:51.058810 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:51.058602    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls podName:1e40077b-94d8-4b82-8982-969b50b1fe96 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:51.558579571 +0000 UTC m=+36.594050720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls") pod "dns-default-jthx2" (UID: "1e40077b-94d8-4b82-8982-969b50b1fe96") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:51.058857 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.058820    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e40077b-94d8-4b82-8982-969b50b1fe96-tmp-dir\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.059047 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.059030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e40077b-94d8-4b82-8982-969b50b1fe96-config-volume\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.062911 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.062890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5739f528-c67f-4c93-807e-80854f5b8b06-cert\") pod \"ingress-canary-qdgdx\" (UID: \"5739f528-c67f-4c93-807e-80854f5b8b06\") " pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:51.067776 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.067744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rg4\" (UniqueName: \"kubernetes.io/projected/1e40077b-94d8-4b82-8982-969b50b1fe96-kube-api-access-v8rg4\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.067892 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.067883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mf6\" (UniqueName: \"kubernetes.io/projected/5739f528-c67f-4c93-807e-80854f5b8b06-kube-api-access-g2mf6\") pod \"ingress-canary-qdgdx\" (UID: \"5739f528-c67f-4c93-807e-80854f5b8b06\") " pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:51.146393 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.146364    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qdgdx"
Sep 09 20:11:51.425681 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.425459    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qdgdx"]
Sep 09 20:11:51.429829 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:11:51.429801    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5739f528_c67f_4c93_807e_80854f5b8b06.slice/crio-7be746bc3b2ae65ad9610627f47edb50e8219563b55366a288088da533583100 WatchSource:0}: Error finding container 7be746bc3b2ae65ad9610627f47edb50e8219563b55366a288088da533583100: Status 404 returned error can't find the container with id 7be746bc3b2ae65ad9610627f47edb50e8219563b55366a288088da533583100
Sep 09 20:11:51.485983 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.485953    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:11:51.485983 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.485979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:11:51.486168 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.485955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:11:51.488884 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.488863    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:11:51.489017 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.488889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:11:51.489017 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.488917    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:11:51.489017 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.488889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:11:51.489195 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.489184    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wr6wr\""
Sep 09 20:11:51.489665 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.489648    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4t9vt\""
Sep 09 20:11:51.562955 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.562928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:51.563120 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:51.563097    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:51.563173 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:51.563167    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls podName:1e40077b-94d8-4b82-8982-969b50b1fe96 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:52.56315207 +0000 UTC m=+37.598623207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls") pod "dns-default-jthx2" (UID: "1e40077b-94d8-4b82-8982-969b50b1fe96") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:51.713302 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.713220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qdgdx" event={"ID":"5739f528-c67f-4c93-807e-80854f5b8b06","Type":"ContainerStarted","Data":"7be746bc3b2ae65ad9610627f47edb50e8219563b55366a288088da533583100"}
Sep 09 20:11:51.716005 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.715981    2566 generic.go:356] "Generic (PLEG): container finished" podID="d69639c4-1be8-4047-a7c4-8c29ad94358b" containerID="765979619f0cd151e86465fd1e7b32b3d2dbf121617218e27c9c05de95c3732c" exitCode=0
Sep 09 20:11:51.716133 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:51.716015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerDied","Data":"765979619f0cd151e86465fd1e7b32b3d2dbf121617218e27c9c05de95c3732c"}
Sep 09 20:11:52.569721 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:52.569683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:52.569920 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:52.569864    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:52.569988 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:52.569937    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls podName:1e40077b-94d8-4b82-8982-969b50b1fe96 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:54.569921357 +0000 UTC m=+39.605392495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls") pod "dns-default-jthx2" (UID: "1e40077b-94d8-4b82-8982-969b50b1fe96") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:52.722896 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:52.722854    2566 generic.go:356] "Generic (PLEG): container finished" podID="d69639c4-1be8-4047-a7c4-8c29ad94358b" containerID="5728196bbbc7cfe565136821f5e82057a4e5327020e215a420b215ccb9c7180d" exitCode=0
Sep 09 20:11:52.723506 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:52.722930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerDied","Data":"5728196bbbc7cfe565136821f5e82057a4e5327020e215a420b215ccb9c7180d"}
Sep 09 20:11:53.728571 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:53.728541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ddscx" event={"ID":"d69639c4-1be8-4047-a7c4-8c29ad94358b","Type":"ContainerStarted","Data":"3f911bdab2028ef882cfa949db9e9ce051e31bc07acfbf862ca833f4e1e045a1"}
Sep 09 20:11:53.751144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:53.751075    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ddscx" podStartSLOduration=10.20650758 podStartE2EDuration="38.751057643s" totalImagesPullingTime="28.544550063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:53.750022887 +0000 UTC m=+38.785494048" watchObservedRunningTime="2026-09-09 20:11:53.751057643 +0000 UTC m=+38.786528798"
Sep 09 20:11:54.588027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:54.587831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:54.588272 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:54.587994    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:54.588272 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:54.588170    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls podName:1e40077b-94d8-4b82-8982-969b50b1fe96 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:58.588147739 +0000 UTC m=+43.623618876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls") pod "dns-default-jthx2" (UID: "1e40077b-94d8-4b82-8982-969b50b1fe96") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:54.732695 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:54.732660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qdgdx" event={"ID":"5739f528-c67f-4c93-807e-80854f5b8b06","Type":"ContainerStarted","Data":"465249c0643ae39db944dc4224935adcc13fa6c5b3c8684ec1d705687ef44833"}
Sep 09 20:11:54.749830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:54.749761    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qdgdx" podStartSLOduration=2.207400628 podStartE2EDuration="4.749738918s" totalImagesPullingTime="2.54233829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:54.748876343 +0000 UTC m=+39.784347501" watchObservedRunningTime="2026-09-09 20:11:54.749738918 +0000 UTC m=+39.785210079"
Sep 09 20:11:58.612962 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:11:58.612919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:11:58.613407 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:58.613057    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:58.613407 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:11:58.613134    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls podName:1e40077b-94d8-4b82-8982-969b50b1fe96 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.613116426 +0000 UTC m=+51.648587564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls") pod "dns-default-jthx2" (UID: "1e40077b-94d8-4b82-8982-969b50b1fe96") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:01.725652 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.725617    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pmzjz"]
Sep 09 20:12:01.771295 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.771255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pmzjz"]
Sep 09 20:12:01.771459 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.771338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz"
Sep 09 20:12:01.773671 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.773649    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gtbm8\""
Sep 09 20:12:01.850191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.850153    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"]
Sep 09 20:12:01.864316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.864294    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d7ff7ffd8-b545h"]
Sep 09 20:12:01.864500 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.864419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:01.867363 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.867316    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-x5b72\""
Sep 09 20:12:01.867363 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.867329    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 20:12:01.867691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.867378    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 20:12:01.867691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.867539    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:12:01.867691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.867629    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 20:12:01.876560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.876541    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"]
Sep 09 20:12:01.876657 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.876624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:01.879350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.879328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 20:12:01.879350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.879329    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 20:12:01.879500 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.879329    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 20:12:01.879697 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.879678    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 20:12:01.879697 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.879690    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lp7w6\""
Sep 09 20:12:01.879821 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.879722    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:12:01.879821 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.879746    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:12:01.890512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.890476    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d7ff7ffd8-b545h"]
Sep 09 20:12:01.936268 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:01.936231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8kj\" (UniqueName: \"kubernetes.io/projected/4348cc4e-6c23-43a3-b1b1-60f0282f9302-kube-api-access-7n8kj\") pod \"network-check-source-685456655f-pmzjz\" (UID: \"4348cc4e-6c23-43a3-b1b1-60f0282f9302\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz"
Sep 09 20:12:02.037135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.037135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qfg\" (UniqueName: \"kubernetes.io/projected/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-kube-api-access-s2qfg\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.037135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-default-certificate\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.037327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-stats-auth\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.037327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e55991d8-4d2d-4d77-8fb0-6959965bc903-config\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.037327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8kj\" (UniqueName: \"kubernetes.io/projected/4348cc4e-6c23-43a3-b1b1-60f0282f9302-kube-api-access-7n8kj\") pod \"network-check-source-685456655f-pmzjz\" (UID: \"4348cc4e-6c23-43a3-b1b1-60f0282f9302\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz"
Sep 09 20:12:02.037327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e55991d8-4d2d-4d77-8fb0-6959965bc903-serving-cert\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.037327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj85w\" (UniqueName: \"kubernetes.io/projected/e55991d8-4d2d-4d77-8fb0-6959965bc903-kube-api-access-zj85w\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.037577 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.037334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.048243 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.048219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8kj\" (UniqueName: \"kubernetes.io/projected/4348cc4e-6c23-43a3-b1b1-60f0282f9302-kube-api-access-7n8kj\") pod \"network-check-source-685456655f-pmzjz\" (UID: \"4348cc4e-6c23-43a3-b1b1-60f0282f9302\") " pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz"
Sep 09 20:12:02.080012 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.079978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz"
Sep 09 20:12:02.138021 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.137982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.138219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.138048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.138219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.138074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qfg\" (UniqueName: \"kubernetes.io/projected/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-kube-api-access-s2qfg\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.138219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.138128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-default-certificate\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.138219 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:02.138202    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:02.638180537 +0000 UTC m=+47.673651675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : configmap references non-existent config key: service-ca.crt
Sep 09 20:12:02.138219 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:02.138212    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:12:02.138495 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:02.138274    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:02.638255956 +0000 UTC m=+47.673727110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : secret "router-metrics-certs-default" not found
Sep 09 20:12:02.138495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.138295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-stats-auth\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.138495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.138367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e55991d8-4d2d-4d77-8fb0-6959965bc903-config\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.138495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.138419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e55991d8-4d2d-4d77-8fb0-6959965bc903-serving-cert\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.138495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.138436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj85w\" (UniqueName: \"kubernetes.io/projected/e55991d8-4d2d-4d77-8fb0-6959965bc903-kube-api-access-zj85w\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.139133 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.139019    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e55991d8-4d2d-4d77-8fb0-6959965bc903-config\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.140659 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.140619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e55991d8-4d2d-4d77-8fb0-6959965bc903-serving-cert\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.140659 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.140647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-default-certificate\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.140779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.140765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-stats-auth\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.148074 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.147819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qfg\" (UniqueName: \"kubernetes.io/projected/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-kube-api-access-s2qfg\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.148818 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.148795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj85w\" (UniqueName: \"kubernetes.io/projected/e55991d8-4d2d-4d77-8fb0-6959965bc903-kube-api-access-zj85w\") pod \"service-ca-operator-859f94c4b-vz6bd\" (UID: \"e55991d8-4d2d-4d77-8fb0-6959965bc903\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.174161 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.174133    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"
Sep 09 20:12:02.373964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.373932    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-pmzjz"]
Sep 09 20:12:02.376855 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:02.376827    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4348cc4e_6c23_43a3_b1b1_60f0282f9302.slice/crio-a70ee52a28127ee601795b0c16320f5eec520c19c8252658c972888c9709e906 WatchSource:0}: Error finding container a70ee52a28127ee601795b0c16320f5eec520c19c8252658c972888c9709e906: Status 404 returned error can't find the container with id a70ee52a28127ee601795b0c16320f5eec520c19c8252658c972888c9709e906
Sep 09 20:12:02.466284 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.466255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd"]
Sep 09 20:12:02.481553 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:02.481525    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55991d8_4d2d_4d77_8fb0_6959965bc903.slice/crio-4d1589fdf23796b6ac10bdfecb781a1ed0393331b3b2a9a83df8e1991fa6c047 WatchSource:0}: Error finding container 4d1589fdf23796b6ac10bdfecb781a1ed0393331b3b2a9a83df8e1991fa6c047: Status 404 returned error can't find the container with id 4d1589fdf23796b6ac10bdfecb781a1ed0393331b3b2a9a83df8e1991fa6c047
Sep 09 20:12:02.643017 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.642919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.643017 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.642970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:02.643239 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:02.643100    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:12:02.643239 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:02.643120    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:03.643101109 +0000 UTC m=+48.678572260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : configmap references non-existent config key: service-ca.crt
Sep 09 20:12:02.643239 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:02.643147    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:03.643135 +0000 UTC m=+48.678606137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : secret "router-metrics-certs-default" not found
Sep 09 20:12:02.753592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.753553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd" event={"ID":"e55991d8-4d2d-4d77-8fb0-6959965bc903","Type":"ContainerStarted","Data":"4d1589fdf23796b6ac10bdfecb781a1ed0393331b3b2a9a83df8e1991fa6c047"}
Sep 09 20:12:02.754636 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:02.754610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz" event={"ID":"4348cc4e-6c23-43a3-b1b1-60f0282f9302","Type":"ContainerStarted","Data":"a70ee52a28127ee601795b0c16320f5eec520c19c8252658c972888c9709e906"}
Sep 09 20:12:03.651281 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:03.651244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:03.651533 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:03.651306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:03.651533 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:03.651421    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:12:03.651533 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:03.651445    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.651421772 +0000 UTC m=+50.686892925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : configmap references non-existent config key: service-ca.crt
Sep 09 20:12:03.651533 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:03.651477    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:05.651465762 +0000 UTC m=+50.686936899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : secret "router-metrics-certs-default" not found
Sep 09 20:12:05.667943 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.667892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:05.668380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.667950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:05.668380 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:05.668077    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.668054676 +0000 UTC m=+54.703525817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : configmap references non-existent config key: service-ca.crt
Sep 09 20:12:05.668380 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:05.668124    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:12:05.668380 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:05.668188    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.668168272 +0000 UTC m=+54.703639414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : secret "router-metrics-certs-default" not found
Sep 09 20:12:05.675740 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.675709    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b7dkr"]
Sep 09 20:12:05.689372 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.689327    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7dkr"]
Sep 09 20:12:05.689561 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.689426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.691762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.691565    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:12:05.692033 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.692016    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:12:05.692343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.692323    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j97b5\""
Sep 09 20:12:05.692901 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.692767    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:12:05.692901 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.692844    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:12:05.765050 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.765010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd" event={"ID":"e55991d8-4d2d-4d77-8fb0-6959965bc903","Type":"ContainerStarted","Data":"0cd300bc47b1020086edc085f5e24cae5479de487cf15e47e638be0da85bfced"}
Sep 09 20:12:05.782880 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.782817    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd" podStartSLOduration=2.313661322 podStartE2EDuration="4.782796725s" totalImagesPullingTime="2.469135403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:05.782411434 +0000 UTC m=+50.817882595" watchObservedRunningTime="2026-09-09 20:12:05.782796725 +0000 UTC m=+50.818267881"
Sep 09 20:12:05.869428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.869368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.869603 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.869483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzs7\" (UniqueName: \"kubernetes.io/projected/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-api-access-qdzs7\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.869682 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.869655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.869799 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.869754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6d164f9-1387-4545-9439-f10ecaa8b1a2-data-volume\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.869927 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.869838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6d164f9-1387-4545-9439-f10ecaa8b1a2-crio-socket\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.971024 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.970926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.971024 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.971021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzs7\" (UniqueName: \"kubernetes.io/projected/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-api-access-qdzs7\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.971295 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.971063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.971295 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.971119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6d164f9-1387-4545-9439-f10ecaa8b1a2-data-volume\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.971295 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.971148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6d164f9-1387-4545-9439-f10ecaa8b1a2-crio-socket\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.971428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.971365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6d164f9-1387-4545-9439-f10ecaa8b1a2-crio-socket\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.971470 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:05.971438    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:05.971502 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:05.971493    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.471475087 +0000 UTC m=+51.506946241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:05.971707 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:05.971681    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:05.971798 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:05.971764    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:06.471744111 +0000 UTC m=+51.507215274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:05.972029 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.972006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6d164f9-1387-4545-9439-f10ecaa8b1a2-data-volume\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:05.982541 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:05.982517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzs7\" (UniqueName: \"kubernetes.io/projected/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-api-access-qdzs7\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:06.475762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:06.475722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:06.475949 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:06.475817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:06.475949 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:06.475875    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:06.475949 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:06.475930    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:06.476069 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:06.475941    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:07.475926798 +0000 UTC m=+52.511397935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:06.476069 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:06.475988    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:07.475976285 +0000 UTC m=+52.511447422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:06.676850 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:06.676814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:12:06.677266 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:06.676967    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:12:06.677266 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:06.677041    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls podName:1e40077b-94d8-4b82-8982-969b50b1fe96 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:22.677021258 +0000 UTC m=+67.712492413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls") pod "dns-default-jthx2" (UID: "1e40077b-94d8-4b82-8982-969b50b1fe96") : secret "dns-default-metrics-tls" not found
Sep 09 20:12:06.769781 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:06.769689    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz" event={"ID":"4348cc4e-6c23-43a3-b1b1-60f0282f9302","Type":"ContainerStarted","Data":"7eaea3c1bc8e653b29958023b1d25a74ddc2c7c7df024e12241d9a8e884139d8"}
Sep 09 20:12:06.786775 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:06.786712    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-pmzjz" podStartSLOduration=1.6085523259999999 podStartE2EDuration="5.786691753s" totalImagesPullingTime="4.178139427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:06.78635864 +0000 UTC m=+51.821829801" watchObservedRunningTime="2026-09-09 20:12:06.786691753 +0000 UTC m=+51.822162906"
Sep 09 20:12:07.483749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:07.483711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:07.483928 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:07.483790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:07.483928 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:07.483867    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:07.483928 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:07.483877    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:07.484025 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:07.483933    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.48391785 +0000 UTC m=+54.519388987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:07.484025 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:07.483948    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:09.483941897 +0000 UTC m=+54.519413034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:07.845206 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:07.845103    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pt8r9_5a62488d-cc14-4591-a50f-1665c9ad2800/dns-node-resolver/0.log"
Sep 09 20:12:08.645330 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.645298    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzvkf_b3ab512a-1904-4284-82da-8a9a0a2aa4e1/node-ca/0.log"
Sep 09 20:12:08.732670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.732640    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"]
Sep 09 20:12:08.735009 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.734994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"
Sep 09 20:12:08.737531 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.737511    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4vrc4\""
Sep 09 20:12:08.737605 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.737527    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 20:12:08.738460 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.738444    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 20:12:08.744404 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.744378    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"]
Sep 09 20:12:08.895593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.895506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6267b\" (UniqueName: \"kubernetes.io/projected/9207419b-8115-4a5d-8c9c-5e10a5f6e52d-kube-api-access-6267b\") pod \"migrator-fdbb8fdff-snf2g\" (UID: \"9207419b-8115-4a5d-8c9c-5e10a5f6e52d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"
Sep 09 20:12:08.996599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:08.996560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6267b\" (UniqueName: \"kubernetes.io/projected/9207419b-8115-4a5d-8c9c-5e10a5f6e52d-kube-api-access-6267b\") pod \"migrator-fdbb8fdff-snf2g\" (UID: \"9207419b-8115-4a5d-8c9c-5e10a5f6e52d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"
Sep 09 20:12:09.007849 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.007791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6267b\" (UniqueName: \"kubernetes.io/projected/9207419b-8115-4a5d-8c9c-5e10a5f6e52d-kube-api-access-6267b\") pod \"migrator-fdbb8fdff-snf2g\" (UID: \"9207419b-8115-4a5d-8c9c-5e10a5f6e52d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"
Sep 09 20:12:09.043676 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.043647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"
Sep 09 20:12:09.310825 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.310685    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g"]
Sep 09 20:12:09.313309 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:09.313281    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9207419b_8115_4a5d_8c9c_5e10a5f6e52d.slice/crio-1fa822f0727b060ed4bcb754deb6d638c2f5aaa54855a47378479e5bb94e89e1 WatchSource:0}: Error finding container 1fa822f0727b060ed4bcb754deb6d638c2f5aaa54855a47378479e5bb94e89e1: Status 404 returned error can't find the container with id 1fa822f0727b060ed4bcb754deb6d638c2f5aaa54855a47378479e5bb94e89e1
Sep 09 20:12:09.446592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.446560    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qdgdx_5739f528-c67f-4c93-807e-80854f5b8b06/serve-healthcheck-canary/0.log"
Sep 09 20:12:09.501116 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.501071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:09.501295 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.501168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:09.501295 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:09.501213    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:09.501295 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:09.501261    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:09.501295 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:09.501276    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:13.501260365 +0000 UTC m=+58.536731505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:09.501484 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:09.501310    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:13.501294448 +0000 UTC m=+58.536765585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:09.702648 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.702609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:09.702829 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.702664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:09.702829 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:09.702791    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:17.702768903 +0000 UTC m=+62.738240060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : configmap references non-existent config key: service-ca.crt
Sep 09 20:12:09.702829 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:09.702818    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:12:09.702958 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:09.702861    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:17.702848832 +0000 UTC m=+62.738319969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : secret "router-metrics-certs-default" not found
Sep 09 20:12:09.778107 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:09.778058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g" event={"ID":"9207419b-8115-4a5d-8c9c-5e10a5f6e52d","Type":"ContainerStarted","Data":"1fa822f0727b060ed4bcb754deb6d638c2f5aaa54855a47378479e5bb94e89e1"}
Sep 09 20:12:10.783259 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:10.783170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g" event={"ID":"9207419b-8115-4a5d-8c9c-5e10a5f6e52d","Type":"ContainerStarted","Data":"642892cec2348d5424b9e49a84b78e498a2f1b2c277e652e7b28b26ddd9245ab"}
Sep 09 20:12:10.783259 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:10.783227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g" event={"ID":"9207419b-8115-4a5d-8c9c-5e10a5f6e52d","Type":"ContainerStarted","Data":"5caafb9efff7c6f9c02454bf2ca336857f6c4eb6bfff05ba0d331156fe6a9557"}
Sep 09 20:12:10.799400 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:10.799349    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-snf2g" podStartSLOduration=1.692179205 podStartE2EDuration="2.799332935s" totalImagesPullingTime="1.10715373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:10.79846023 +0000 UTC m=+55.833931390" watchObservedRunningTime="2026-09-09 20:12:10.799332935 +0000 UTC m=+55.834804090"
Sep 09 20:12:13.531221 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:13.531185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:13.531673 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:13.531261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:13.531673 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:13.531303    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:12:13.531673 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:13.531355    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:13.531673 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:13.531378    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:21.531361487 +0000 UTC m=+66.566832624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : configmap "kube-rbac-proxy" not found
Sep 09 20:12:13.531673 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:13.531408    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:21.53139511 +0000 UTC m=+66.566866247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:15.702687 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:15.702662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h7z6h"
Sep 09 20:12:17.758639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:17.758601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:17.759054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:17.758659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:17.759054 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:17.758766    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle podName:681323ae-d7bd-43ce-8bc6-34b06c1df3b8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:33.758745892 +0000 UTC m=+78.794217043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle") pod "router-default-7d7ff7ffd8-b545h" (UID: "681323ae-d7bd-43ce-8bc6-34b06c1df3b8") : configmap references non-existent config key: service-ca.crt
Sep 09 20:12:17.761159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:17.761127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-metrics-certs\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:21.184461 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.184422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:12:21.187397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.187376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:21.197627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.197602    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d7c169-a7a9-4978-88ba-1a166aa21947-metrics-certs\") pod \"network-metrics-daemon-qhxc4\" (UID: \"38d7c169-a7a9-4978-88ba-1a166aa21947\") " pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:12:21.204196 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.204174    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4t9vt\""
Sep 09 20:12:21.212290 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.212268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qhxc4"
Sep 09 20:12:21.285480 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.285420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:12:21.285655 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.285531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:12:21.288144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.288115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr2m\" (UniqueName: \"kubernetes.io/projected/ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef-kube-api-access-pmr2m\") pod \"network-check-target-2vsjc\" (UID: \"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef\") " pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:12:21.288325 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.288304    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:21.298593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.298566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78e54e15-216a-4f30-ac4e-b1848d33090b-original-pull-secret\") pod \"global-pull-secret-syncer-tq7t4\" (UID: \"78e54e15-216a-4f30-ac4e-b1848d33090b\") " pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:12:21.495562 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.495492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tq7t4"
Sep 09 20:12:21.508350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.508325    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wr6wr\""
Sep 09 20:12:21.511519 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.511452    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qhxc4"]
Sep 09 20:12:21.515852 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.515831    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:12:21.516189 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:21.516169    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d7c169_a7a9_4978_88ba_1a166aa21947.slice/crio-19a3d01e2e28d581517518340a78bfc229a0418e8992decdf997cc20f4e8aa3e WatchSource:0}: Error finding container 19a3d01e2e28d581517518340a78bfc229a0418e8992decdf997cc20f4e8aa3e: Status 404 returned error can't find the container with id 19a3d01e2e28d581517518340a78bfc229a0418e8992decdf997cc20f4e8aa3e
Sep 09 20:12:21.587761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.587731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:21.588006 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.587987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:21.588154 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:21.588133    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:21.588330 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:21.588317    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls podName:e6d164f9-1387-4545-9439-f10ecaa8b1a2 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:37.588299485 +0000 UTC m=+82.623770623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7dkr" (UID: "e6d164f9-1387-4545-9439-f10ecaa8b1a2") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:12:21.588430 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.588362    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6d164f9-1387-4545-9439-f10ecaa8b1a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:21.797538 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.797465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tq7t4"]
Sep 09 20:12:21.800637 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:21.800609    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e54e15_216a_4f30_ac4e_b1848d33090b.slice/crio-35bb323f489f2b449051df570b74691a6565c2970c87d04e01ade66efe134121 WatchSource:0}: Error finding container 35bb323f489f2b449051df570b74691a6565c2970c87d04e01ade66efe134121: Status 404 returned error can't find the container with id 35bb323f489f2b449051df570b74691a6565c2970c87d04e01ade66efe134121
Sep 09 20:12:21.812386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.812351    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tq7t4" event={"ID":"78e54e15-216a-4f30-ac4e-b1848d33090b","Type":"ContainerStarted","Data":"35bb323f489f2b449051df570b74691a6565c2970c87d04e01ade66efe134121"}
Sep 09 20:12:21.813572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.813550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhxc4" event={"ID":"38d7c169-a7a9-4978-88ba-1a166aa21947","Type":"ContainerStarted","Data":"19a3d01e2e28d581517518340a78bfc229a0418e8992decdf997cc20f4e8aa3e"}
Sep 09 20:12:21.830318 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:21.830226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vsjc"]
Sep 09 20:12:21.832740 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:21.832710    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2c3ee4_6828_4da8_afa0_1c9ada0d67ef.slice/crio-456c4db724ebaff9c1a34a0cd287831759eccdcb3aba1192a3702f1d173e836b WatchSource:0}: Error finding container 456c4db724ebaff9c1a34a0cd287831759eccdcb3aba1192a3702f1d173e836b: Status 404 returned error can't find the container with id 456c4db724ebaff9c1a34a0cd287831759eccdcb3aba1192a3702f1d173e836b
Sep 09 20:12:22.696803 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.696767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:12:22.699505 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.699483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e40077b-94d8-4b82-8982-969b50b1fe96-metrics-tls\") pod \"dns-default-jthx2\" (UID: \"1e40077b-94d8-4b82-8982-969b50b1fe96\") " pod="openshift-dns/dns-default-jthx2"
Sep 09 20:12:22.819554 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.819521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhxc4" event={"ID":"38d7c169-a7a9-4978-88ba-1a166aa21947","Type":"ContainerStarted","Data":"377ac534b8fdd9cd69ea858583c265b2bfb5b3b1ab3f084fb48ae97473fa543f"}
Sep 09 20:12:22.821135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.821108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vsjc" event={"ID":"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef","Type":"ContainerStarted","Data":"7fa1267505ed9d0569e12f8a49523ba5b6e344c9d8568eec5b83739070f63c6d"}
Sep 09 20:12:22.821259 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.821156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vsjc" event={"ID":"ae2c3ee4-6828-4da8-afa0-1c9ada0d67ef","Type":"ContainerStarted","Data":"456c4db724ebaff9c1a34a0cd287831759eccdcb3aba1192a3702f1d173e836b"}
Sep 09 20:12:22.821427 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.821405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:12:22.836614 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.836569    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2vsjc" podStartSLOduration=67.836552904 podStartE2EDuration="1m7.836552904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:22.835575918 +0000 UTC m=+67.871047076" watchObservedRunningTime="2026-09-09 20:12:22.836552904 +0000 UTC m=+67.872024063"
Sep 09 20:12:22.944522 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.944492    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftsch\""
Sep 09 20:12:22.952020 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:22.951940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jthx2"
Sep 09 20:12:23.221865 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:23.221733    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jthx2"]
Sep 09 20:12:23.226877 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:23.226848    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e40077b_94d8_4b82_8982_969b50b1fe96.slice/crio-b09fd19e84359cae8ee0f55f2ca21db20560451d85209c496cbe60779f4d2b98 WatchSource:0}: Error finding container b09fd19e84359cae8ee0f55f2ca21db20560451d85209c496cbe60779f4d2b98: Status 404 returned error can't find the container with id b09fd19e84359cae8ee0f55f2ca21db20560451d85209c496cbe60779f4d2b98
Sep 09 20:12:23.826834 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:23.826797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qhxc4" event={"ID":"38d7c169-a7a9-4978-88ba-1a166aa21947","Type":"ContainerStarted","Data":"433b50f22cffb084cbb2611819f46d4e83cb24ea08a45dbb930209ffbcc3fced"}
Sep 09 20:12:23.828141 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:23.828111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jthx2" event={"ID":"1e40077b-94d8-4b82-8982-969b50b1fe96","Type":"ContainerStarted","Data":"b09fd19e84359cae8ee0f55f2ca21db20560451d85209c496cbe60779f4d2b98"}
Sep 09 20:12:23.844099 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:23.844017    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qhxc4" podStartSLOduration=67.661607898 podStartE2EDuration="1m8.843998275s" totalImagesPullingTime="1.182390377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:23.84320177 +0000 UTC m=+68.878672971" watchObservedRunningTime="2026-09-09 20:12:23.843998275 +0000 UTC m=+68.879469436"
Sep 09 20:12:26.842523 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:26.842486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tq7t4" event={"ID":"78e54e15-216a-4f30-ac4e-b1848d33090b","Type":"ContainerStarted","Data":"9d7ba792ec05cf29a5d0d67a1c7f8621c9d97560bd3077c642856750db238499"}
Sep 09 20:12:26.846005 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:26.845974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jthx2" event={"ID":"1e40077b-94d8-4b82-8982-969b50b1fe96","Type":"ContainerStarted","Data":"9c694bcb23393e8764fa6b6fc93992bbc7026e3da1cb09049a1f4264e19b790b"}
Sep 09 20:12:26.857335 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:26.857282    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tq7t4" podStartSLOduration=66.931962956 podStartE2EDuration="1m11.857265875s" totalImagesPullingTime="4.925302919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.856758074 +0000 UTC m=+71.892229234" watchObservedRunningTime="2026-09-09 20:12:26.857265875 +0000 UTC m=+71.892737033"
Sep 09 20:12:27.851052 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:27.851020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jthx2" event={"ID":"1e40077b-94d8-4b82-8982-969b50b1fe96","Type":"ContainerStarted","Data":"5e620debb20dabb6b094233ee01496cb1e17d83bb20c95d335bd9d5552c6b1be"}
Sep 09 20:12:27.871095 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:27.871026    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jthx2" podStartSLOduration=34.383138222 podStartE2EDuration="37.87100941s" totalImagesPullingTime="3.487871188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:27.870307743 +0000 UTC m=+72.905778949" watchObservedRunningTime="2026-09-09 20:12:27.87100941 +0000 UTC m=+72.906480571"
Sep 09 20:12:28.854536 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:28.854495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jthx2"
Sep 09 20:12:29.404383 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.404348    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"]
Sep 09 20:12:29.409058 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.409039    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.412257 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.412237    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m5kg4\""
Sep 09 20:12:29.413478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.413434    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:12:29.413588 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.413432    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:12:29.413588 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.413440    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:12:29.413588 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.413462    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 20:12:29.423358 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.423327    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"]
Sep 09 20:12:29.453673 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.453640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/538eada8-814a-4dab-aa16-5e4cde47481b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs\" (UID: \"538eada8-814a-4dab-aa16-5e4cde47481b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.453673 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.453674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlpj\" (UniqueName: \"kubernetes.io/projected/538eada8-814a-4dab-aa16-5e4cde47481b-kube-api-access-6qlpj\") pod \"managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs\" (UID: \"538eada8-814a-4dab-aa16-5e4cde47481b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.457441 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.457407    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-869956756f-v882x"]
Sep 09 20:12:29.461335 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.461319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.464188 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.464170    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:12:29.464298 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.464192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:12:29.464515 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.464497    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gslg8\""
Sep 09 20:12:29.464550 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.464518    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:12:29.469383 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.469367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:12:29.472682 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.472663    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869956756f-v882x"]
Sep 09 20:12:29.554584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5fb96632-7127-4e2a-b7af-285de927da6d-registry-certificates\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.554584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5fb96632-7127-4e2a-b7af-285de927da6d-installation-pull-secrets\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.554796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5fb96632-7127-4e2a-b7af-285de927da6d-image-registry-private-configuration\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.554796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-registry-tls\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.554796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlpj\" (UniqueName: \"kubernetes.io/projected/538eada8-814a-4dab-aa16-5e4cde47481b-kube-api-access-6qlpj\") pod \"managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs\" (UID: \"538eada8-814a-4dab-aa16-5e4cde47481b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.554796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/538eada8-814a-4dab-aa16-5e4cde47481b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs\" (UID: \"538eada8-814a-4dab-aa16-5e4cde47481b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.554796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pww2p\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-kube-api-access-pww2p\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.554796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-bound-sa-token\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.554796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5fb96632-7127-4e2a-b7af-285de927da6d-ca-trust-extracted\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.555119 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.554842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fb96632-7127-4e2a-b7af-285de927da6d-trusted-ca\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.557356 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.557330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/538eada8-814a-4dab-aa16-5e4cde47481b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs\" (UID: \"538eada8-814a-4dab-aa16-5e4cde47481b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.565964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.565937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlpj\" (UniqueName: \"kubernetes.io/projected/538eada8-814a-4dab-aa16-5e4cde47481b-kube-api-access-6qlpj\") pod \"managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs\" (UID: \"538eada8-814a-4dab-aa16-5e4cde47481b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.655202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fb96632-7127-4e2a-b7af-285de927da6d-trusted-ca\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5fb96632-7127-4e2a-b7af-285de927da6d-registry-certificates\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5fb96632-7127-4e2a-b7af-285de927da6d-installation-pull-secrets\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5fb96632-7127-4e2a-b7af-285de927da6d-image-registry-private-configuration\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-registry-tls\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pww2p\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-kube-api-access-pww2p\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-bound-sa-token\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5fb96632-7127-4e2a-b7af-285de927da6d-ca-trust-extracted\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.655854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.655828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5fb96632-7127-4e2a-b7af-285de927da6d-ca-trust-extracted\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.656356 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.656331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5fb96632-7127-4e2a-b7af-285de927da6d-trusted-ca\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.656442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.656369    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5fb96632-7127-4e2a-b7af-285de927da6d-registry-certificates\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.657878 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.657860    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-registry-tls\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.657961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.657887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5fb96632-7127-4e2a-b7af-285de927da6d-installation-pull-secrets\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.658060 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.658038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5fb96632-7127-4e2a-b7af-285de927da6d-image-registry-private-configuration\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.665216 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.665194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-bound-sa-token\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.665306 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.665281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pww2p\" (UniqueName: \"kubernetes.io/projected/5fb96632-7127-4e2a-b7af-285de927da6d-kube-api-access-pww2p\") pod \"image-registry-869956756f-v882x\" (UID: \"5fb96632-7127-4e2a-b7af-285de927da6d\") " pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:29.729131 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.729096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"
Sep 09 20:12:29.770094 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:29.770042    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:30.001159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:30.001065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs"]
Sep 09 20:12:30.004315 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:30.004293    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod538eada8_814a_4dab_aa16_5e4cde47481b.slice/crio-5950dc68e5168b49b826c30aa0c87919cfe12554e4232467fba261eaaec7a277 WatchSource:0}: Error finding container 5950dc68e5168b49b826c30aa0c87919cfe12554e4232467fba261eaaec7a277: Status 404 returned error can't find the container with id 5950dc68e5168b49b826c30aa0c87919cfe12554e4232467fba261eaaec7a277
Sep 09 20:12:30.046715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:30.046685    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869956756f-v882x"]
Sep 09 20:12:30.049300 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:30.049277    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb96632_7127_4e2a_b7af_285de927da6d.slice/crio-5f1c3b6e88ad8c65e14f4e7aaed492cd9e897b600cd134c4096f6469e35381b3 WatchSource:0}: Error finding container 5f1c3b6e88ad8c65e14f4e7aaed492cd9e897b600cd134c4096f6469e35381b3: Status 404 returned error can't find the container with id 5f1c3b6e88ad8c65e14f4e7aaed492cd9e897b600cd134c4096f6469e35381b3
Sep 09 20:12:30.861952 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:30.861914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869956756f-v882x" event={"ID":"5fb96632-7127-4e2a-b7af-285de927da6d","Type":"ContainerStarted","Data":"80c339a6b4251cc30ad4e825a8bf1e36185da0dea2331eee387afe5d92be1f43"}
Sep 09 20:12:30.862145 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:30.861963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-869956756f-v882x" event={"ID":"5fb96632-7127-4e2a-b7af-285de927da6d","Type":"ContainerStarted","Data":"5f1c3b6e88ad8c65e14f4e7aaed492cd9e897b600cd134c4096f6469e35381b3"}
Sep 09 20:12:30.862145 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:30.861998    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:30.863203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:30.863179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs" event={"ID":"538eada8-814a-4dab-aa16-5e4cde47481b","Type":"ContainerStarted","Data":"5950dc68e5168b49b826c30aa0c87919cfe12554e4232467fba261eaaec7a277"}
Sep 09 20:12:30.881999 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:30.881946    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-869956756f-v882x" podStartSLOduration=1.881931674 podStartE2EDuration="1.881931674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:30.880726174 +0000 UTC m=+75.916197332" watchObservedRunningTime="2026-09-09 20:12:30.881931674 +0000 UTC m=+75.917402828"
Sep 09 20:12:33.787339 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:33.787276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:33.787921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:33.787897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681323ae-d7bd-43ce-8bc6-34b06c1df3b8-service-ca-bundle\") pod \"router-default-7d7ff7ffd8-b545h\" (UID: \"681323ae-d7bd-43ce-8bc6-34b06c1df3b8\") " pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:33.877365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:33.877326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs" event={"ID":"538eada8-814a-4dab-aa16-5e4cde47481b","Type":"ContainerStarted","Data":"5f4a10279ac81d5363a302f82bcedea6cb39c6a84ddc1687849fe1531c2fc5c8"}
Sep 09 20:12:33.893156 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:33.893109    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65b4f7ddc6-nw8rs" podStartSLOduration=2.077970831 podStartE2EDuration="4.893079344s" totalImagesPullingTime="2.815108513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:33.892048681 +0000 UTC m=+78.927519840" watchObservedRunningTime="2026-09-09 20:12:33.893079344 +0000 UTC m=+78.928550502"
Sep 09 20:12:33.988000 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:33.987971    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lp7w6\""
Sep 09 20:12:33.995459 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:33.995436    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:34.276435 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:34.276408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d7ff7ffd8-b545h"]
Sep 09 20:12:34.278792 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:34.278767    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681323ae_d7bd_43ce_8bc6_34b06c1df3b8.slice/crio-b8d458d67b383950b2f161734571297025c7567cc81a0875e0228acb1d8e4060 WatchSource:0}: Error finding container b8d458d67b383950b2f161734571297025c7567cc81a0875e0228acb1d8e4060: Status 404 returned error can't find the container with id b8d458d67b383950b2f161734571297025c7567cc81a0875e0228acb1d8e4060
Sep 09 20:12:34.882160 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:34.882121    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h" event={"ID":"681323ae-d7bd-43ce-8bc6-34b06c1df3b8","Type":"ContainerStarted","Data":"cd83d8e4637bb81f11da0c696280b63cb1f0386b09efa4d445fab694f1eb80db"}
Sep 09 20:12:34.882613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:34.882180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h" event={"ID":"681323ae-d7bd-43ce-8bc6-34b06c1df3b8","Type":"ContainerStarted","Data":"b8d458d67b383950b2f161734571297025c7567cc81a0875e0228acb1d8e4060"}
Sep 09 20:12:34.903762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:34.903707    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h" podStartSLOduration=33.903691037 podStartE2EDuration="33.903691037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:34.902647125 +0000 UTC m=+79.938118285" watchObservedRunningTime="2026-09-09 20:12:34.903691037 +0000 UTC m=+79.939162197"
Sep 09 20:12:34.996656 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:34.996612    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:34.999222 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:34.999199    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:35.884979 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:35.884945    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:35.886140 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:35.886118    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d7ff7ffd8-b545h"
Sep 09 20:12:37.617998 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:37.617883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:37.620336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:37.620311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6d164f9-1387-4545-9439-f10ecaa8b1a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7dkr\" (UID: \"e6d164f9-1387-4545-9439-f10ecaa8b1a2\") " pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:37.806019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:37.805986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j97b5\""
Sep 09 20:12:37.813933 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:37.813908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7dkr"
Sep 09 20:12:38.084099 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:38.084055    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7dkr"]
Sep 09 20:12:38.086617 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:38.086587    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d164f9_1387_4545_9439_f10ecaa8b1a2.slice/crio-c4e2bbd898e7dfffcc3ec65c8db48b2876b56b747c8a1986c99b917a9365d72e WatchSource:0}: Error finding container c4e2bbd898e7dfffcc3ec65c8db48b2876b56b747c8a1986c99b917a9365d72e: Status 404 returned error can't find the container with id c4e2bbd898e7dfffcc3ec65c8db48b2876b56b747c8a1986c99b917a9365d72e
Sep 09 20:12:38.859639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:38.859611    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jthx2"
Sep 09 20:12:38.897283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:38.897253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7dkr" event={"ID":"e6d164f9-1387-4545-9439-f10ecaa8b1a2","Type":"ContainerStarted","Data":"c54b0f607fafd5aef197eeb8201ffd70310cf9d967723bd3dc92853bfae5107a"}
Sep 09 20:12:38.897395 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:38.897312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7dkr" event={"ID":"e6d164f9-1387-4545-9439-f10ecaa8b1a2","Type":"ContainerStarted","Data":"c4e2bbd898e7dfffcc3ec65c8db48b2876b56b747c8a1986c99b917a9365d72e"}
Sep 09 20:12:39.902514 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:39.902479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7dkr" event={"ID":"e6d164f9-1387-4545-9439-f10ecaa8b1a2","Type":"ContainerStarted","Data":"17739c077cbe7659ecd6f003012daa39f99145d3142dd7748a696f0865377bc3"}
Sep 09 20:12:40.908162 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:40.908126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7dkr" event={"ID":"e6d164f9-1387-4545-9439-f10ecaa8b1a2","Type":"ContainerStarted","Data":"7cfa3dab8de1949018ecd1c82ac57e3376358f800706a2f81bdb5b31a1c2c017"}
Sep 09 20:12:40.926817 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:40.926725    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b7dkr" podStartSLOduration=33.47665488 podStartE2EDuration="35.926710551s" totalImagesPullingTime="2.450055671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:40.925257571 +0000 UTC m=+85.960728731" watchObservedRunningTime="2026-09-09 20:12:40.926710551 +0000 UTC m=+85.962181710"
Sep 09 20:12:47.175718 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.175675    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"]
Sep 09 20:12:47.180046 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.180018    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.182881 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.182859    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:12:47.184063 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.184031    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:12:47.184063 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.184032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l8wcf\""
Sep 09 20:12:47.184247 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.184074    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:12:47.184247 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.184032    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:12:47.184247 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.184058    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:12:47.185169 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.185151    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"]
Sep 09 20:12:47.187206 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.187190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.189249 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.189232    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 20:12:47.189696 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.189679    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:12:47.189756 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.189698    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s4rl6\""
Sep 09 20:12:47.189756 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.189683    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 20:12:47.192823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.192802    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bbs5f"]
Sep 09 20:12:47.195191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.195173    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"]
Sep 09 20:12:47.195389 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.195358    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.198150 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.198126    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:12:47.198251 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.198167    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:12:47.198251 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.198223    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9nvqc\""
Sep 09 20:12:47.198251 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.198239    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:12:47.205078 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.205056    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"]
Sep 09 20:12:47.282942 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.282905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrd9t\" (UniqueName: \"kubernetes.io/projected/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-api-access-zrd9t\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.282942 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.282942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-tls\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.282969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ef352ed3-6351-47a1-98f6-998e9902b2fa-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.283202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.282991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-sys\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.283202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-textfile\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.283202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-wtmp\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-root\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvp7\" (UniqueName: \"kubernetes.io/projected/0480caf2-ce19-4d3a-830d-30a27f5854ef-kube-api-access-5lvp7\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.283397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.283397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a0be3-99f1-41a9-8119-927574ac4f68-metrics-client-ca\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.283397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0480caf2-ce19-4d3a-830d-30a27f5854ef-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.283397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5cs\" (UniqueName: \"kubernetes.io/projected/9b0a0be3-99f1-41a9-8119-927574ac4f68-kube-api-access-wg5cs\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef352ed3-6351-47a1-98f6-998e9902b2fa-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.283593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.283593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.283593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.283493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.384647 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.384647 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0480caf2-ce19-4d3a-830d-30a27f5854ef-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.384889 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5cs\" (UniqueName: \"kubernetes.io/projected/9b0a0be3-99f1-41a9-8119-927574ac4f68-kube-api-access-wg5cs\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.384889 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef352ed3-6351-47a1-98f6-998e9902b2fa-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.384889 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.384889 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.384889 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrd9t\" (UniqueName: \"kubernetes.io/projected/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-api-access-zrd9t\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.385157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-tls\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.384988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ef352ed3-6351-47a1-98f6-998e9902b2fa-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.385157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-sys\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.385157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-textfile\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385157 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.385517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-wtmp\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-root\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-sys\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvp7\" (UniqueName: \"kubernetes.io/projected/0480caf2-ce19-4d3a-830d-30a27f5854ef-kube-api-access-5lvp7\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.385517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.385517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a0be3-99f1-41a9-8119-927574ac4f68-metrics-client-ca\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef352ed3-6351-47a1-98f6-998e9902b2fa-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.385789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0480caf2-ce19-4d3a-830d-30a27f5854ef-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.385789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-accelerators-collector-config\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-wtmp\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.385789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385785    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ef352ed3-6351-47a1-98f6-998e9902b2fa-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.386051 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385800    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b0a0be3-99f1-41a9-8119-927574ac4f68-root\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.386051 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.385852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a0be3-99f1-41a9-8119-927574ac4f68-metrics-client-ca\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.386051 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:47.385868    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:12:47.386051 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:47.385921    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-tls podName:9b0a0be3-99f1-41a9-8119-927574ac4f68 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:47.885902228 +0000 UTC m=+92.921373370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-tls") pod "node-exporter-bbs5f" (UID: "9b0a0be3-99f1-41a9-8119-927574ac4f68") : secret "node-exporter-tls" not found
Sep 09 20:12:47.386456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.386106    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-textfile\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.386456 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:47.386210    2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 09 20:12:47.386456 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:12:47.386275    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-tls podName:0480caf2-ce19-4d3a-830d-30a27f5854ef nodeName:}" failed. No retries permitted until 2026-09-09 20:12:47.886257104 +0000 UTC m=+92.921728245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-wx6pv" (UID: "0480caf2-ce19-4d3a-830d-30a27f5854ef") : secret "openshift-state-metrics-tls" not found
Sep 09 20:12:47.386456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.386389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.387601 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.387576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.387695 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.387641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.388136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.388109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.388395 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.388375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.395990 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.395963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5cs\" (UniqueName: \"kubernetes.io/projected/9b0a0be3-99f1-41a9-8119-927574ac4f68-kube-api-access-wg5cs\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.396717 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.396694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrd9t\" (UniqueName: \"kubernetes.io/projected/ef352ed3-6351-47a1-98f6-998e9902b2fa-kube-api-access-zrd9t\") pod \"kube-state-metrics-74cb57df7-6wspw\" (UID: \"ef352ed3-6351-47a1-98f6-998e9902b2fa\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.396815 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.396794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvp7\" (UniqueName: \"kubernetes.io/projected/0480caf2-ce19-4d3a-830d-30a27f5854ef-kube-api-access-5lvp7\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.498848 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.498777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"
Sep 09 20:12:47.774550 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.774438    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6wspw"]
Sep 09 20:12:47.777056 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:47.777025    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef352ed3_6351_47a1_98f6_998e9902b2fa.slice/crio-647a5ccbfd992d13415d1625aa25a3f831fe0698791ee29181c9c47fb0d23520 WatchSource:0}: Error finding container 647a5ccbfd992d13415d1625aa25a3f831fe0698791ee29181c9c47fb0d23520: Status 404 returned error can't find the container with id 647a5ccbfd992d13415d1625aa25a3f831fe0698791ee29181c9c47fb0d23520
Sep 09 20:12:47.889309 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.889262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-tls\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.889508 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.889369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.891823 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.891798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b0a0be3-99f1-41a9-8119-927574ac4f68-node-exporter-tls\") pod \"node-exporter-bbs5f\" (UID: \"9b0a0be3-99f1-41a9-8119-927574ac4f68\") " pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:47.891912 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.891848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0480caf2-ce19-4d3a-830d-30a27f5854ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wx6pv\" (UID: \"0480caf2-ce19-4d3a-830d-30a27f5854ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:47.931925 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:47.931888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw" event={"ID":"ef352ed3-6351-47a1-98f6-998e9902b2fa","Type":"ContainerStarted","Data":"647a5ccbfd992d13415d1625aa25a3f831fe0698791ee29181c9c47fb0d23520"}
Sep 09 20:12:48.090101 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.089998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"
Sep 09 20:12:48.104449 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.104424    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bbs5f"
Sep 09 20:12:48.113626 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:48.113597    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0a0be3_99f1_41a9_8119_927574ac4f68.slice/crio-7fb8f2c803e14a55777e57f25da91c082ae183fecea44f1bd23f4c817b93960b WatchSource:0}: Error finding container 7fb8f2c803e14a55777e57f25da91c082ae183fecea44f1bd23f4c817b93960b: Status 404 returned error can't find the container with id 7fb8f2c803e14a55777e57f25da91c082ae183fecea44f1bd23f4c817b93960b
Sep 09 20:12:48.148053 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.148026    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:12:48.153324 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.153301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.156246 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156222    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:12:48.156363 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156250    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:12:48.156363 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156222    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:12:48.156605 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156592    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:12:48.156605 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156597    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f25v2\""
Sep 09 20:12:48.156821 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156800    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:12:48.156905 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156818    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:12:48.156905 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156836    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:12:48.156905 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156799    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:12:48.157052 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.156802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:12:48.167147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.167120    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:12:48.229549 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.229503    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-r92mr"]
Sep 09 20:12:48.231850 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.231830    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-r92mr"
Sep 09 20:12:48.237146 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.234936    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:12:48.237146 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.235131    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h92gw\""
Sep 09 20:12:48.237721 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.237699    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:12:48.244604 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.244582    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-r92mr"]
Sep 09 20:12:48.293002 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.292962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-volume\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293194 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293194 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293194 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxbp\" (UniqueName: \"kubernetes.io/projected/823113ee-0dba-419e-bb10-80c0eb4f0a92-kube-api-access-lfxbp\") pod \"downloads-85d98c998f-r92mr\" (UID: \"823113ee-0dba-419e-bb10-80c0eb4f0a92\") " pod="openshift-console/downloads-85d98c998f-r92mr"
Sep 09 20:12:48.293194 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-web-config\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-out\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pd84\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-kube-api-access-5pd84\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.293595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.293558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.366871 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.366693    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv"]
Sep 09 20:12:48.369352 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:48.369320    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0480caf2_ce19_4d3a_830d_30a27f5854ef.slice/crio-c092479f5b972b36424bbe2b8d25157eef980a83456f22ec3692ef6ec1f14b98 WatchSource:0}: Error finding container c092479f5b972b36424bbe2b8d25157eef980a83456f22ec3692ef6ec1f14b98: Status 404 returned error can't find the container with id c092479f5b972b36424bbe2b8d25157eef980a83456f22ec3692ef6ec1f14b98
Sep 09 20:12:48.394648 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.394760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.394760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pd84\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-kube-api-access-5pd84\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.394832 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.394832 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.394895 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-volume\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.394895 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.394991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.394900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.395149 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.395128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.396922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.396412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxbp\" (UniqueName: \"kubernetes.io/projected/823113ee-0dba-419e-bb10-80c0eb4f0a92-kube-api-access-lfxbp\") pod \"downloads-85d98c998f-r92mr\" (UID: \"823113ee-0dba-419e-bb10-80c0eb4f0a92\") " pod="openshift-console/downloads-85d98c998f-r92mr"
Sep 09 20:12:48.396922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.396501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.396922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.396506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-web-config\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.396922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.396574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.396922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.396675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.396922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.396755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.396922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.396785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-out\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.398851 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.398321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.398851 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.398412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.400510 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.400040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.400510 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.400437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.400656 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.400612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-volume\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.401355 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.401318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.401665 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.401623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-web-config\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.402011 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.401968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.402113 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.402048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-out\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.402237 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.402198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.404857 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.404835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pd84\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-kube-api-access-5pd84\") pod \"alertmanager-main-0\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.405408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.405388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxbp\" (UniqueName: \"kubernetes.io/projected/823113ee-0dba-419e-bb10-80c0eb4f0a92-kube-api-access-lfxbp\") pod \"downloads-85d98c998f-r92mr\" (UID: \"823113ee-0dba-419e-bb10-80c0eb4f0a92\") " pod="openshift-console/downloads-85d98c998f-r92mr"
Sep 09 20:12:48.465178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.465148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:12:48.546820 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.546409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-r92mr"
Sep 09 20:12:48.767199 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.767172    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:12:48.840682 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.840601    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-r92mr"]
Sep 09 20:12:48.937805 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.937720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbs5f" event={"ID":"9b0a0be3-99f1-41a9-8119-927574ac4f68","Type":"ContainerStarted","Data":"7fb8f2c803e14a55777e57f25da91c082ae183fecea44f1bd23f4c817b93960b"}
Sep 09 20:12:48.940252 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.940218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv" event={"ID":"0480caf2-ce19-4d3a-830d-30a27f5854ef","Type":"ContainerStarted","Data":"c0ed57d746b26a36f897a0cb5e94ccd4070ff6d283da31ec1885f630471e7ec8"}
Sep 09 20:12:48.940417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.940275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv" event={"ID":"0480caf2-ce19-4d3a-830d-30a27f5854ef","Type":"ContainerStarted","Data":"0cecfffef15e42526c33ad6590fb88c9907d7e844c7da0d5b49a5eb8fc87b80d"}
Sep 09 20:12:48.940417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:48.940303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv" event={"ID":"0480caf2-ce19-4d3a-830d-30a27f5854ef","Type":"ContainerStarted","Data":"c092479f5b972b36424bbe2b8d25157eef980a83456f22ec3692ef6ec1f14b98"}
Sep 09 20:12:48.997982 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:48.997943    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8011e5c1_df64_4ea3_a90a_e7fe133b6ead.slice/crio-7e7b1fa130a89c6409aace18c7d4c8965c0a825d212e012f57c8cf5e59122d63 WatchSource:0}: Error finding container 7e7b1fa130a89c6409aace18c7d4c8965c0a825d212e012f57c8cf5e59122d63: Status 404 returned error can't find the container with id 7e7b1fa130a89c6409aace18c7d4c8965c0a825d212e012f57c8cf5e59122d63
Sep 09 20:12:48.998581 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:12:48.998560    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823113ee_0dba_419e_bb10_80c0eb4f0a92.slice/crio-c4fd19702c12a220a1553ad0e13827ac212f1794466ea11f310aea89fdbdf618 WatchSource:0}: Error finding container c4fd19702c12a220a1553ad0e13827ac212f1794466ea11f310aea89fdbdf618: Status 404 returned error can't find the container with id c4fd19702c12a220a1553ad0e13827ac212f1794466ea11f310aea89fdbdf618
Sep 09 20:12:49.947866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.947830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw" event={"ID":"ef352ed3-6351-47a1-98f6-998e9902b2fa","Type":"ContainerStarted","Data":"a8ce30f176b2735061de9ca01e6e57003862461d2fd062d2b7dc87514c94816a"}
Sep 09 20:12:49.948329 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.947892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw" event={"ID":"ef352ed3-6351-47a1-98f6-998e9902b2fa","Type":"ContainerStarted","Data":"648b2e324538753a9abf71f57b003494f314f7cfb1410d9f9a98ca1a3ece7eb4"}
Sep 09 20:12:49.948329 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.947911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw" event={"ID":"ef352ed3-6351-47a1-98f6-998e9902b2fa","Type":"ContainerStarted","Data":"68fa9a199e9b5bd430782543829d5d4f7a503c68905f70ca58f3c9a6e5c94c30"}
Sep 09 20:12:49.950384 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.950353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerStarted","Data":"7e7b1fa130a89c6409aace18c7d4c8965c0a825d212e012f57c8cf5e59122d63"}
Sep 09 20:12:49.952713 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.952684    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b0a0be3-99f1-41a9-8119-927574ac4f68" containerID="1da249455ac8d3f3238266eb4fd95bb563a43034dfac2910dac45b27da31898d" exitCode=0
Sep 09 20:12:49.953026 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.952962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbs5f" event={"ID":"9b0a0be3-99f1-41a9-8119-927574ac4f68","Type":"ContainerDied","Data":"1da249455ac8d3f3238266eb4fd95bb563a43034dfac2910dac45b27da31898d"}
Sep 09 20:12:49.954456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.954432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-r92mr" event={"ID":"823113ee-0dba-419e-bb10-80c0eb4f0a92","Type":"ContainerStarted","Data":"c4fd19702c12a220a1553ad0e13827ac212f1794466ea11f310aea89fdbdf618"}
Sep 09 20:12:49.972826 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:49.971526    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6wspw" podStartSLOduration=1.496711356 podStartE2EDuration="2.971506414s" totalImagesPullingTime="1.474795058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:49.969564749 +0000 UTC m=+95.005035907" watchObservedRunningTime="2026-09-09 20:12:49.971506414 +0000 UTC m=+95.006977576"
Sep 09 20:12:50.960177 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:50.960139    2566 generic.go:356] "Generic (PLEG): container finished" podID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerID="f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c" exitCode=0
Sep 09 20:12:50.960627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:50.960228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c"}
Sep 09 20:12:50.963293 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:50.963269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbs5f" event={"ID":"9b0a0be3-99f1-41a9-8119-927574ac4f68","Type":"ContainerStarted","Data":"4ed83cb2fef83943f140c7bb935c92c2678051992a12cecf1f112ee8b387620c"}
Sep 09 20:12:50.963384 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:50.963322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bbs5f" event={"ID":"9b0a0be3-99f1-41a9-8119-927574ac4f68","Type":"ContainerStarted","Data":"46dbf9541b420788f233afafa675d395469c897fd8ceb713f9dee81e465bb1b0"}
Sep 09 20:12:50.966524 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:50.966500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv" event={"ID":"0480caf2-ce19-4d3a-830d-30a27f5854ef","Type":"ContainerStarted","Data":"c396a0387c06e8ad3f897d8538e1afb6810441eee22dfd3b32ea7ad02c2e6473"}
Sep 09 20:12:51.006007 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:51.005950    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wx6pv" podStartSLOduration=2.199212954 podStartE2EDuration="4.005931788s" totalImagesPullingTime="1.806718834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:51.004424272 +0000 UTC m=+96.039895431" watchObservedRunningTime="2026-09-09 20:12:51.005931788 +0000 UTC m=+96.041402947"
Sep 09 20:12:51.023004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:51.022942    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bbs5f" podStartSLOduration=2.776244265 podStartE2EDuration="4.022919779s" totalImagesPullingTime="1.140428388s" totalInitContainerRuntime="106.247126ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:51.02187516 +0000 UTC m=+96.057346322" watchObservedRunningTime="2026-09-09 20:12:51.022919779 +0000 UTC m=+96.058390940"
Sep 09 20:12:51.874554 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:51.874500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-869956756f-v882x"
Sep 09 20:12:52.979412 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:52.979369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerStarted","Data":"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"}
Sep 09 20:12:52.979790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:52.979470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerStarted","Data":"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"}
Sep 09 20:12:52.979790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:52.979519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerStarted","Data":"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"}
Sep 09 20:12:52.979790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:52.979557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerStarted","Data":"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"}
Sep 09 20:12:52.979790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:52.979613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerStarted","Data":"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"}
Sep 09 20:12:53.831295 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:53.831196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2vsjc"
Sep 09 20:12:53.989044 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:53.989012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerStarted","Data":"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"}
Sep 09 20:12:54.017226 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:12:54.017164    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.129934614 podStartE2EDuration="6.017140942s" totalImagesPullingTime="3.860717896s" totalInitContainerRuntime="26.488432ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:54.014252616 +0000 UTC m=+99.049723778" watchObservedRunningTime="2026-09-09 20:12:54.017140942 +0000 UTC m=+99.052612102"
Sep 09 20:13:06.045890 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:06.045684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-r92mr" event={"ID":"823113ee-0dba-419e-bb10-80c0eb4f0a92","Type":"ContainerStarted","Data":"e5833ceef6b3e73637bb5e3d2b5e43141aa0f02c8807d7dc8d15f844ed4f44e5"}
Sep 09 20:13:06.048386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:06.046671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-r92mr"
Sep 09 20:13:06.052206 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:06.052178    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-r92mr"
Sep 09 20:13:06.066222 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:06.066161    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-r92mr" podStartSLOduration=1.639395302 podStartE2EDuration="18.066140522s" totalImagesPullingTime="16.42674522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:06.064592142 +0000 UTC m=+111.100063319" watchObservedRunningTime="2026-09-09 20:13:06.066140522 +0000 UTC m=+111.101611682"
Sep 09 20:13:11.070027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:11.069988    2566 generic.go:356] "Generic (PLEG): container finished" podID="e55991d8-4d2d-4d77-8fb0-6959965bc903" containerID="0cd300bc47b1020086edc085f5e24cae5479de487cf15e47e638be0da85bfced" exitCode=0
Sep 09 20:13:11.070567 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:11.070072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd" event={"ID":"e55991d8-4d2d-4d77-8fb0-6959965bc903","Type":"ContainerDied","Data":"0cd300bc47b1020086edc085f5e24cae5479de487cf15e47e638be0da85bfced"}
Sep 09 20:13:11.070567 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:11.070453    2566 scope.go:117] "RemoveContainer" containerID="0cd300bc47b1020086edc085f5e24cae5479de487cf15e47e638be0da85bfced"
Sep 09 20:13:12.077319 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:13:12.077276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-vz6bd" event={"ID":"e55991d8-4d2d-4d77-8fb0-6959965bc903","Type":"ContainerStarted","Data":"6563fe39ecf251e50f75acede7f1a5048b0735c41cfdd0e91f95643d4cdefa1e"}
Sep 09 20:14:07.457356 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:07.457271    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:14:07.457866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:07.457610    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="alertmanager" containerID="cri-o://3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3" gracePeriod=120
Sep 09 20:14:07.457866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:07.457668    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-metric" containerID="cri-o://226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2" gracePeriod=120
Sep 09 20:14:07.457866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:07.457701    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="config-reloader" containerID="cri-o://d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97" gracePeriod=120
Sep 09 20:14:07.457866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:07.457679    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-web" containerID="cri-o://8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837" gracePeriod=120
Sep 09 20:14:07.457866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:07.457765    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="prom-label-proxy" containerID="cri-o://d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf" gracePeriod=120
Sep 09 20:14:07.457866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:07.457726    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy" containerID="cri-o://b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d" gracePeriod=120
Sep 09 20:14:08.298392 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298349    2566 generic.go:356] "Generic (PLEG): container finished" podID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerID="d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf" exitCode=0
Sep 09 20:14:08.298392 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298377    2566 generic.go:356] "Generic (PLEG): container finished" podID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerID="b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d" exitCode=0
Sep 09 20:14:08.298392 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298384    2566 generic.go:356] "Generic (PLEG): container finished" podID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerID="d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97" exitCode=0
Sep 09 20:14:08.298392 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298390    2566 generic.go:356] "Generic (PLEG): container finished" podID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerID="3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3" exitCode=0
Sep 09 20:14:08.298662 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"}
Sep 09 20:14:08.298662 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"}
Sep 09 20:14:08.298662 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"}
Sep 09 20:14:08.298662 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.298565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"}
Sep 09 20:14:08.707437 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.707409    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:08.871555 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-cluster-tls-config\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871734 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871572    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-volume\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871734 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871599    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-main-tls\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871734 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-main-db\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871734 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-tls-assets\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871734 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-web-config\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871771    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-out\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-web\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-trusted-ca-bundle\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871851    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pd84\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-kube-api-access-5pd84\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-metrics-client-ca\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.871976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.871904    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\" (UID: \"8011e5c1-df64-4ea3-a90a-e7fe133b6ead\") "
Sep 09 20:14:08.872497 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.872167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:08.873077 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.872628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:08.874914 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.874878    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:08.875365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.875124    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:08.875476 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.875357    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:08.875546 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.875475    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:08.875603 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.875536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:08.875658 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.875622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-out" (OuterVolumeSpecName: "config-out") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:08.875841 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.875799    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:08.875841 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.875818    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-volume" (OuterVolumeSpecName: "config-volume") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:08.877394 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.877368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-kube-api-access-5pd84" (OuterVolumeSpecName: "kube-api-access-5pd84") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "kube-api-access-5pd84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:08.878895 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.878874    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:08.887101 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.887056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-web-config" (OuterVolumeSpecName: "web-config") pod "8011e5c1-df64-4ea3-a90a-e7fe133b6ead" (UID: "8011e5c1-df64-4ea3-a90a-e7fe133b6ead"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:08.973140 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973058    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973140 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973135    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-web-config\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973140 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973150    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-out\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973162    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973175    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973186    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pd84\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-kube-api-access-5pd84\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973199    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-metrics-client-ca\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973210    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973222    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-cluster-tls-config\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973233    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-config-volume\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973245    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-secret-alertmanager-main-tls\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973264    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-alertmanager-main-db\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:08.973414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:08.973275    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8011e5c1-df64-4ea3-a90a-e7fe133b6ead-tls-assets\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:14:09.305889 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.305798    2566 generic.go:356] "Generic (PLEG): container finished" podID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerID="226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2" exitCode=0
Sep 09 20:14:09.305889 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.305825    2566 generic.go:356] "Generic (PLEG): container finished" podID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerID="8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837" exitCode=0
Sep 09 20:14:09.306135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.305890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"}
Sep 09 20:14:09.306135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.305906    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.306135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.305973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"}
Sep 09 20:14:09.306135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.306018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8011e5c1-df64-4ea3-a90a-e7fe133b6ead","Type":"ContainerDied","Data":"7e7b1fa130a89c6409aace18c7d4c8965c0a825d212e012f57c8cf5e59122d63"}
Sep 09 20:14:09.306135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.306038    2566 scope.go:117] "RemoveContainer" containerID="d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"
Sep 09 20:14:09.314976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.314955    2566 scope.go:117] "RemoveContainer" containerID="226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"
Sep 09 20:14:09.321784 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.321764    2566 scope.go:117] "RemoveContainer" containerID="b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"
Sep 09 20:14:09.328470 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.328443    2566 scope.go:117] "RemoveContainer" containerID="8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"
Sep 09 20:14:09.330744 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.330694    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:14:09.335037 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.335015    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:14:09.336574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.336558    2566 scope.go:117] "RemoveContainer" containerID="d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"
Sep 09 20:14:09.343807 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.343789    2566 scope.go:117] "RemoveContainer" containerID="3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"
Sep 09 20:14:09.350873 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.350853    2566 scope.go:117] "RemoveContainer" containerID="f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c"
Sep 09 20:14:09.358056 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.358041    2566 scope.go:117] "RemoveContainer" containerID="d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"
Sep 09 20:14:09.358345 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:14:09.358326    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf\": container with ID starting with d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf not found: ID does not exist" containerID="d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"
Sep 09 20:14:09.358390 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.358354    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"} err="failed to get container status \"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf\": rpc error: code = NotFound desc = could not find container \"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf\": container with ID starting with d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf not found: ID does not exist"
Sep 09 20:14:09.358390 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.358385    2566 scope.go:117] "RemoveContainer" containerID="226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"
Sep 09 20:14:09.358631 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:14:09.358615    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2\": container with ID starting with 226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2 not found: ID does not exist" containerID="226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"
Sep 09 20:14:09.358666 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.358635    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"} err="failed to get container status \"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2\": rpc error: code = NotFound desc = could not find container \"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2\": container with ID starting with 226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2 not found: ID does not exist"
Sep 09 20:14:09.358666 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.358648    2566 scope.go:117] "RemoveContainer" containerID="b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"
Sep 09 20:14:09.358845 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:14:09.358830    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d\": container with ID starting with b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d not found: ID does not exist" containerID="b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"
Sep 09 20:14:09.358885 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.358849    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"} err="failed to get container status \"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d\": rpc error: code = NotFound desc = could not find container \"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d\": container with ID starting with b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d not found: ID does not exist"
Sep 09 20:14:09.358885 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.358860    2566 scope.go:117] "RemoveContainer" containerID="8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"
Sep 09 20:14:09.359060 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:14:09.359043    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837\": container with ID starting with 8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837 not found: ID does not exist" containerID="8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"
Sep 09 20:14:09.359129 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359064    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"} err="failed to get container status \"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837\": rpc error: code = NotFound desc = could not find container \"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837\": container with ID starting with 8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837 not found: ID does not exist"
Sep 09 20:14:09.359129 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359077    2566 scope.go:117] "RemoveContainer" containerID="d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"
Sep 09 20:14:09.359303 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:14:09.359289    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97\": container with ID starting with d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97 not found: ID does not exist" containerID="d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"
Sep 09 20:14:09.359339 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359307    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"} err="failed to get container status \"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97\": rpc error: code = NotFound desc = could not find container \"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97\": container with ID starting with d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97 not found: ID does not exist"
Sep 09 20:14:09.359339 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359323    2566 scope.go:117] "RemoveContainer" containerID="3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"
Sep 09 20:14:09.359510 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:14:09.359494    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3\": container with ID starting with 3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3 not found: ID does not exist" containerID="3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"
Sep 09 20:14:09.359549 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359513    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"} err="failed to get container status \"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3\": rpc error: code = NotFound desc = could not find container \"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3\": container with ID starting with 3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3 not found: ID does not exist"
Sep 09 20:14:09.359549 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359525    2566 scope.go:117] "RemoveContainer" containerID="f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c"
Sep 09 20:14:09.359738 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:14:09.359719    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c\": container with ID starting with f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c not found: ID does not exist" containerID="f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c"
Sep 09 20:14:09.359795 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359747    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c"} err="failed to get container status \"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c\": rpc error: code = NotFound desc = could not find container \"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c\": container with ID starting with f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c not found: ID does not exist"
Sep 09 20:14:09.359795 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359769    2566 scope.go:117] "RemoveContainer" containerID="d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"
Sep 09 20:14:09.359992 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359976    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf"} err="failed to get container status \"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf\": rpc error: code = NotFound desc = could not find container \"d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf\": container with ID starting with d668a5821e9b72d3396bb9c65b53630f66fa35650ddb8b245aa0b7be6c64b2cf not found: ID does not exist"
Sep 09 20:14:09.360038 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.359992    2566 scope.go:117] "RemoveContainer" containerID="226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"
Sep 09 20:14:09.360198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360182    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2"} err="failed to get container status \"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2\": rpc error: code = NotFound desc = could not find container \"226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2\": container with ID starting with 226e773556e0bb2c01d6fb34bc4a03d450603c906859f41e841e865731d368b2 not found: ID does not exist"
Sep 09 20:14:09.360236 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360198    2566 scope.go:117] "RemoveContainer" containerID="b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"
Sep 09 20:14:09.360418 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360401    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d"} err="failed to get container status \"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d\": rpc error: code = NotFound desc = could not find container \"b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d\": container with ID starting with b214da926febfc4dc7c55163ddfce3674a65c5ae5b5c7d261e7d66530634e29d not found: ID does not exist"
Sep 09 20:14:09.360478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360419    2566 scope.go:117] "RemoveContainer" containerID="8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"
Sep 09 20:14:09.360670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360655    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837"} err="failed to get container status \"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837\": rpc error: code = NotFound desc = could not find container \"8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837\": container with ID starting with 8dbbe94b1ebc94284af951d171821d36ed52198fe611c1b19945f0acf5c60837 not found: ID does not exist"
Sep 09 20:14:09.360722 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360672    2566 scope.go:117] "RemoveContainer" containerID="d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"
Sep 09 20:14:09.360867 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360850    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97"} err="failed to get container status \"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97\": rpc error: code = NotFound desc = could not find container \"d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97\": container with ID starting with d7bca1f63cfdbecc48006f967e912b7a1aa07e59f1d75a1c782d6bc48ac30a97 not found: ID does not exist"
Sep 09 20:14:09.360906 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.360867    2566 scope.go:117] "RemoveContainer" containerID="3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"
Sep 09 20:14:09.361068 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.361047    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3"} err="failed to get container status \"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3\": rpc error: code = NotFound desc = could not find container \"3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3\": container with ID starting with 3328899bd06009d1aad4863011192112cde2cacca5bfda705d536630550ea2e3 not found: ID does not exist"
Sep 09 20:14:09.361132 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.361071    2566 scope.go:117] "RemoveContainer" containerID="f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c"
Sep 09 20:14:09.361291 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.361276    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c"} err="failed to get container status \"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c\": rpc error: code = NotFound desc = could not find container \"f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c\": container with ID starting with f808667517c86b92e4d6e8a9b8523c4b91b2e9b32b957e62be0897421f71278c not found: ID does not exist"
Sep 09 20:14:09.364834 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.364815    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:14:09.365078 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365067    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="alertmanager"
Sep 09 20:14:09.365139 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365101    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="alertmanager"
Sep 09 20:14:09.365139 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365117    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="init-config-reloader"
Sep 09 20:14:09.365139 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365127    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="init-config-reloader"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365141    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-web"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365149    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-web"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365160    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="prom-label-proxy"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365165    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="prom-label-proxy"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365172    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365177    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365187    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="config-reloader"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365193    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="config-reloader"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365200    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-metric"
Sep 09 20:14:09.365227 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365205    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-metric"
Sep 09 20:14:09.365512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365250    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-web"
Sep 09 20:14:09.365512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365264    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="prom-label-proxy"
Sep 09 20:14:09.365512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365272    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="config-reloader"
Sep 09 20:14:09.365512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365279    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="alertmanager"
Sep 09 20:14:09.365512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365284    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy"
Sep 09 20:14:09.365512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.365289    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" containerName="kube-rbac-proxy-metric"
Sep 09 20:14:09.368672 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.368658    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.370975 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.370959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:14:09.371033 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.370959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:14:09.371336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.371303    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:14:09.371336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.371320    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:14:09.371493 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.371354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:14:09.371493 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.371440    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:14:09.371493 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.371457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:14:09.371700 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.371683    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f25v2\""
Sep 09 20:14:09.371788 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.371770    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:14:09.381527 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.381496    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:14:09.386055 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.386029    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:14:09.476407 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476407 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9a68be8-3474-4e0d-bb91-2d3b2b635122-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-config-volume\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9a68be8-3474-4e0d-bb91-2d3b2b635122-config-out\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a68be8-3474-4e0d-bb91-2d3b2b635122-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-web-config\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9a68be8-3474-4e0d-bb91-2d3b2b635122-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9a68be8-3474-4e0d-bb91-2d3b2b635122-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.476888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.476788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnq4\" (UniqueName: \"kubernetes.io/projected/d9a68be8-3474-4e0d-bb91-2d3b2b635122-kube-api-access-rlnq4\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.490103 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.490059    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8011e5c1-df64-4ea3-a90a-e7fe133b6ead" path="/var/lib/kubelet/pods/8011e5c1-df64-4ea3-a90a-e7fe133b6ead/volumes"
Sep 09 20:14:09.577595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-config-volume\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.577595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9a68be8-3474-4e0d-bb91-2d3b2b635122-config-out\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.577595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.577784 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.577784 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.577784 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a68be8-3474-4e0d-bb91-2d3b2b635122-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.577922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-web-config\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.577922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.577852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.579044 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.579018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9a68be8-3474-4e0d-bb91-2d3b2b635122-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.579254 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.579233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9a68be8-3474-4e0d-bb91-2d3b2b635122-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.579410 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.579394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9a68be8-3474-4e0d-bb91-2d3b2b635122-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.579519 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.579505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnq4\" (UniqueName: \"kubernetes.io/projected/d9a68be8-3474-4e0d-bb91-2d3b2b635122-kube-api-access-rlnq4\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.579675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.579658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.579779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.579764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9a68be8-3474-4e0d-bb91-2d3b2b635122-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.580857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-config-volume\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.580960    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.581135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.581134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d9a68be8-3474-4e0d-bb91-2d3b2b635122-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.581163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-web-config\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581605 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.581419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581657 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.581635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581878 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.581653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9a68be8-3474-4e0d-bb91-2d3b2b635122-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.581961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.581936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9a68be8-3474-4e0d-bb91-2d3b2b635122-config-out\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.582222 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.582203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9a68be8-3474-4e0d-bb91-2d3b2b635122-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.583431 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.583407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d9a68be8-3474-4e0d-bb91-2d3b2b635122-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.590310 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.590291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnq4\" (UniqueName: \"kubernetes.io/projected/d9a68be8-3474-4e0d-bb91-2d3b2b635122-kube-api-access-rlnq4\") pod \"alertmanager-main-0\" (UID: \"d9a68be8-3474-4e0d-bb91-2d3b2b635122\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.684412 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.684385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:14:09.964857 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:09.964682    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:14:09.967483 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:14:09.967452    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a68be8_3474_4e0d_bb91_2d3b2b635122.slice/crio-edaac210f4ed5ce30a2b671caa5b411d2baa7395d2a1ef2ad6e93a86c796cf37 WatchSource:0}: Error finding container edaac210f4ed5ce30a2b671caa5b411d2baa7395d2a1ef2ad6e93a86c796cf37: Status 404 returned error can't find the container with id edaac210f4ed5ce30a2b671caa5b411d2baa7395d2a1ef2ad6e93a86c796cf37
Sep 09 20:14:10.312960 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:10.312871    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9a68be8-3474-4e0d-bb91-2d3b2b635122" containerID="fe0f305f0162d4c2b160f98b1d71dc419fb0836c5ee7459a2f092244581d0ab2" exitCode=0
Sep 09 20:14:10.312960 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:10.312926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerDied","Data":"fe0f305f0162d4c2b160f98b1d71dc419fb0836c5ee7459a2f092244581d0ab2"}
Sep 09 20:14:10.313162 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:10.312976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerStarted","Data":"edaac210f4ed5ce30a2b671caa5b411d2baa7395d2a1ef2ad6e93a86c796cf37"}
Sep 09 20:14:11.321133 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.321077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerStarted","Data":"ffd881b0e6307f13c75c4adc36fc52c3440b5c6f6f0ea3bfdbf648ebc6b3c835"}
Sep 09 20:14:11.321631 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.321176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerStarted","Data":"736547a128dfad7d6b5ea26e7f8aa2a5702019a72b5a6b99c5e7d108f247288f"}
Sep 09 20:14:11.321631 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.321221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerStarted","Data":"83ff3750cc3fcfc00aa80b10fa2b82751dc92148b72d5c09a6b3d430fcfbc540"}
Sep 09 20:14:11.321631 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.321269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerStarted","Data":"9ebab9dc87dae133a598ea325763b1b165c6547ffd730bbbc509848b965ed871"}
Sep 09 20:14:11.321631 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.321320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerStarted","Data":"f22bb07aabf293396b3eb94039258eb1fac26e0c7d3f231f2259a94bedf5f66e"}
Sep 09 20:14:11.321631 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.321358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d9a68be8-3474-4e0d-bb91-2d3b2b635122","Type":"ContainerStarted","Data":"18bc073532807d90f2dd0db9c0fca25618192c6108828223861f8716d196a6cb"}
Sep 09 20:14:11.358768 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.358711    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.342558709 podStartE2EDuration="2.358694679s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.13597ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:11.35553838 +0000 UTC m=+176.391009562" watchObservedRunningTime="2026-09-09 20:14:11.358694679 +0000 UTC m=+176.394165853"
Sep 09 20:14:11.495777 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.495746    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55d88bb879-jvq2g"]
Sep 09 20:14:11.498034 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.498017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.501406 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.501383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 20:14:11.501516 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.501443    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 20:14:11.501516 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.501489    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 20:14:11.501614 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.501447    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n58d4\""
Sep 09 20:14:11.501614 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.501567    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 20:14:11.501695 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.501640    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 20:14:11.507525 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.507508    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 20:14:11.517661 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.517633    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55d88bb879-jvq2g"]
Sep 09 20:14:11.594502 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-telemeter-client-tls\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.594502 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-metrics-client-ca\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.594502 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.594502 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg94q\" (UniqueName: \"kubernetes.io/projected/535c62cd-c525-45c8-894c-159be2a18b7d-kube-api-access-xg94q\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.594724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-serving-certs-ca-bundle\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.594724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-secret-telemeter-client\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.594724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-federate-client-tls\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.594724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.594643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.694968 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.694934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695105 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.694973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg94q\" (UniqueName: \"kubernetes.io/projected/535c62cd-c525-45c8-894c-159be2a18b7d-kube-api-access-xg94q\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695105 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.695026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-serving-certs-ca-bundle\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.695109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-secret-telemeter-client\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.695150    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-federate-client-tls\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.695188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.695223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-telemeter-client-tls\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.695247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-metrics-client-ca\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.695929 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.695903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-serving-certs-ca-bundle\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.696301 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.696099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-metrics-client-ca\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.696456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.696429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535c62cd-c525-45c8-894c-159be2a18b7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.697868 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.697838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.697868 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.697843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-telemeter-client-tls\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.697984 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.697904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-federate-client-tls\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.698019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.697981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/535c62cd-c525-45c8-894c-159be2a18b7d-secret-telemeter-client\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.706396 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.706372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg94q\" (UniqueName: \"kubernetes.io/projected/535c62cd-c525-45c8-894c-159be2a18b7d-kube-api-access-xg94q\") pod \"telemeter-client-55d88bb879-jvq2g\" (UID: \"535c62cd-c525-45c8-894c-159be2a18b7d\") " pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:11.807488 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:11.807454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g"
Sep 09 20:14:12.090221 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:12.090197    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55d88bb879-jvq2g"]
Sep 09 20:14:12.092041 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:14:12.092013    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535c62cd_c525_45c8_894c_159be2a18b7d.slice/crio-0a76b07c67e1da29e2ec58812e0691fcc6ca6fd4a0cf53eb1ad0fdfed3cf6498 WatchSource:0}: Error finding container 0a76b07c67e1da29e2ec58812e0691fcc6ca6fd4a0cf53eb1ad0fdfed3cf6498: Status 404 returned error can't find the container with id 0a76b07c67e1da29e2ec58812e0691fcc6ca6fd4a0cf53eb1ad0fdfed3cf6498
Sep 09 20:14:12.326773 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:12.326733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g" event={"ID":"535c62cd-c525-45c8-894c-159be2a18b7d","Type":"ContainerStarted","Data":"0a76b07c67e1da29e2ec58812e0691fcc6ca6fd4a0cf53eb1ad0fdfed3cf6498"}
Sep 09 20:14:14.336691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:14.336588    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g" event={"ID":"535c62cd-c525-45c8-894c-159be2a18b7d","Type":"ContainerStarted","Data":"bf512eb3065ac344894d24e974f682d11d47eebd8b902c6dd386a4afbc285ebd"}
Sep 09 20:14:14.336691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:14.336650    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g" event={"ID":"535c62cd-c525-45c8-894c-159be2a18b7d","Type":"ContainerStarted","Data":"bedba8b70fbacc69a8e0ff5c4de9f2a48f4807148070bfd1f4ae013ac95b802f"}
Sep 09 20:14:14.336691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:14.336672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g" event={"ID":"535c62cd-c525-45c8-894c-159be2a18b7d","Type":"ContainerStarted","Data":"26566bd072fc876bb65b1c344c2887478855009bd745bf09640e58220dba5496"}
Sep 09 20:14:14.365604 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:14:14.365547    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55d88bb879-jvq2g" podStartSLOduration=1.5873549150000001 podStartE2EDuration="3.365527601s" totalImagesPullingTime="1.778172686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:14.36443343 +0000 UTC m=+179.399904589" watchObservedRunningTime="2026-09-09 20:14:14.365527601 +0000 UTC m=+179.400998804"
Sep 09 20:16:15.396337 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:16:15.396301    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:17:33.405786 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.405707    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"]
Sep 09 20:17:33.408856 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.408840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.411662 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.411597    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:17:33.411662 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.411599    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:17:33.411662 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.411647    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:17:33.411945 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.411650    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:17:33.412685 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.412666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:17:33.412805 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.412788    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hwkjt\""
Sep 09 20:17:33.417949 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.417925    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"]
Sep 09 20:17:33.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.496251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdtsv\" (UniqueName: \"kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-kube-api-access-jdtsv\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.496286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.496283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.496481 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.496359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-certificates\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.596739 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.596706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-certificates\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.596946 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.596799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtsv\" (UniqueName: \"kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-kube-api-access-jdtsv\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.596946 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.596828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.596946 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:17:33.596861    2566 secret.go:281] references non-existent secret key: tls.crt
Sep 09 20:17:33.596946 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:17:33.596886    2566 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 09 20:17:33.596946 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:17:33.596909    2566 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh: references non-existent secret key: tls.crt
Sep 09 20:17:33.597279 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:17:33.596980    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-certificates podName:cd9b7e64-a219-4db3-a4ce-a4b77419ae90 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:34.096958636 +0000 UTC m=+379.132429791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-certificates") pod "keda-metrics-apiserver-77857b7b98-pb6sh" (UID: "cd9b7e64-a219-4db3-a4ce-a4b77419ae90") : references non-existent secret key: tls.crt
Sep 09 20:17:33.597279 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.597241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:33.607951 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:33.607931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdtsv\" (UniqueName: \"kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-kube-api-access-jdtsv\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:34.101751 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:34.101711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-certificates\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:34.104364 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:34.104333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd9b7e64-a219-4db3-a4ce-a4b77419ae90-certificates\") pod \"keda-metrics-apiserver-77857b7b98-pb6sh\" (UID: \"cd9b7e64-a219-4db3-a4ce-a4b77419ae90\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:34.320286 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:34.320245    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:34.592651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:34.592617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"]
Sep 09 20:17:34.596167 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:17:34.596140    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9b7e64_a219_4db3_a4ce_a4b77419ae90.slice/crio-2cf93ee895aa252c78b49e467cc30a33d0be4a65f517263e43647f3f82ebf735 WatchSource:0}: Error finding container 2cf93ee895aa252c78b49e467cc30a33d0be4a65f517263e43647f3f82ebf735: Status 404 returned error can't find the container with id 2cf93ee895aa252c78b49e467cc30a33d0be4a65f517263e43647f3f82ebf735
Sep 09 20:17:34.597696 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:34.597679    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:17:35.115059 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:35.115015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh" event={"ID":"cd9b7e64-a219-4db3-a4ce-a4b77419ae90","Type":"ContainerStarted","Data":"2cf93ee895aa252c78b49e467cc30a33d0be4a65f517263e43647f3f82ebf735"}
Sep 09 20:17:39.134971 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:39.134934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh" event={"ID":"cd9b7e64-a219-4db3-a4ce-a4b77419ae90","Type":"ContainerStarted","Data":"57bdcd1f3b247d868fa3b46c53efedc1b72fda36b522fdd16335bed528843f7c"}
Sep 09 20:17:39.135415 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:39.135071    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:17:39.153105 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:39.153032    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh" podStartSLOduration=2.297179985 podStartE2EDuration="6.153012732s" totalImagesPullingTime="3.855832747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:39.15198273 +0000 UTC m=+384.187453891" watchObservedRunningTime="2026-09-09 20:17:39.153012732 +0000 UTC m=+384.188483892"
Sep 09 20:17:50.146337 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:17:50.146305    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-pb6sh"
Sep 09 20:18:44.535231 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.535152    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"]
Sep 09 20:18:44.538415 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.538394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:44.541452 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.541415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hnm46\""
Sep 09 20:18:44.541627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.541473    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 20:18:44.541627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.541500    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:18:44.554421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.554397    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"]
Sep 09 20:18:44.612454 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.607764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2be95aa4-1150-467b-a578-4cce5245d408-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-84c5p\" (UID: \"2be95aa4-1150-467b-a578-4cce5245d408\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:44.612454 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.607845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqxj\" (UniqueName: \"kubernetes.io/projected/2be95aa4-1150-467b-a578-4cce5245d408-kube-api-access-hqqxj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-84c5p\" (UID: \"2be95aa4-1150-467b-a578-4cce5245d408\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:44.708639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.708596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2be95aa4-1150-467b-a578-4cce5245d408-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-84c5p\" (UID: \"2be95aa4-1150-467b-a578-4cce5245d408\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:44.708792 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.708645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqxj\" (UniqueName: \"kubernetes.io/projected/2be95aa4-1150-467b-a578-4cce5245d408-kube-api-access-hqqxj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-84c5p\" (UID: \"2be95aa4-1150-467b-a578-4cce5245d408\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:44.709050 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.709028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2be95aa4-1150-467b-a578-4cce5245d408-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-84c5p\" (UID: \"2be95aa4-1150-467b-a578-4cce5245d408\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:44.720546 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.720511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqxj\" (UniqueName: \"kubernetes.io/projected/2be95aa4-1150-467b-a578-4cce5245d408-kube-api-access-hqqxj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-84c5p\" (UID: \"2be95aa4-1150-467b-a578-4cce5245d408\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:44.847821 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:44.847710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"
Sep 09 20:18:45.128304 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:45.128274    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p"]
Sep 09 20:18:45.129891 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:18:45.129868    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be95aa4_1150_467b_a578_4cce5245d408.slice/crio-94d90726b4b328f6dd4320602862bbdfc5024a3f09bb1c9f92c77a80916d8cdd WatchSource:0}: Error finding container 94d90726b4b328f6dd4320602862bbdfc5024a3f09bb1c9f92c77a80916d8cdd: Status 404 returned error can't find the container with id 94d90726b4b328f6dd4320602862bbdfc5024a3f09bb1c9f92c77a80916d8cdd
Sep 09 20:18:45.401498 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:45.401411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p" event={"ID":"2be95aa4-1150-467b-a578-4cce5245d408","Type":"ContainerStarted","Data":"94d90726b4b328f6dd4320602862bbdfc5024a3f09bb1c9f92c77a80916d8cdd"}
Sep 09 20:18:48.421022 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:48.420074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p" event={"ID":"2be95aa4-1150-467b-a578-4cce5245d408","Type":"ContainerStarted","Data":"d53d79fcb61e5b3a273bacf9b8785bbd4ad9b804c789874b0ac70388e130b4bb"}
Sep 09 20:18:48.444308 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:18:48.444252    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-84c5p" podStartSLOduration=1.860923549 podStartE2EDuration="4.444232564s" totalImagesPullingTime="2.583309015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:48.441177397 +0000 UTC m=+453.476648558" watchObservedRunningTime="2026-09-09 20:18:48.444232564 +0000 UTC m=+453.479703724"
Sep 09 20:19:12.810645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.810609    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"]
Sep 09 20:19:12.813852 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.813835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:12.817427 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.817408    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:19:12.819287 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.819255    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 20:19:12.819387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.819295    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hn8kd\""
Sep 09 20:19:12.826315 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.826295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"]
Sep 09 20:19:12.950436 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.950406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vr8z\" (UniqueName: \"kubernetes.io/projected/f386a589-3e94-4453-a100-b49fc67952f1-kube-api-access-7vr8z\") pod \"openshift-lws-operator-fd8ccff4c-5fq7q\" (UID: \"f386a589-3e94-4453-a100-b49fc67952f1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:12.950436 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:12.950449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f386a589-3e94-4453-a100-b49fc67952f1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5fq7q\" (UID: \"f386a589-3e94-4453-a100-b49fc67952f1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:13.051145 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:13.051076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vr8z\" (UniqueName: \"kubernetes.io/projected/f386a589-3e94-4453-a100-b49fc67952f1-kube-api-access-7vr8z\") pod \"openshift-lws-operator-fd8ccff4c-5fq7q\" (UID: \"f386a589-3e94-4453-a100-b49fc67952f1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:13.051342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:13.051169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f386a589-3e94-4453-a100-b49fc67952f1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5fq7q\" (UID: \"f386a589-3e94-4453-a100-b49fc67952f1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:13.051556 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:13.051534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f386a589-3e94-4453-a100-b49fc67952f1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5fq7q\" (UID: \"f386a589-3e94-4453-a100-b49fc67952f1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:13.061175 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:13.061121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vr8z\" (UniqueName: \"kubernetes.io/projected/f386a589-3e94-4453-a100-b49fc67952f1-kube-api-access-7vr8z\") pod \"openshift-lws-operator-fd8ccff4c-5fq7q\" (UID: \"f386a589-3e94-4453-a100-b49fc67952f1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:13.122528 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:13.122486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"
Sep 09 20:19:13.399728 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:13.399484    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q"]
Sep 09 20:19:13.402965 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:19:13.402926    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf386a589_3e94_4453_a100_b49fc67952f1.slice/crio-fa18c494daecec9b3a2c505b8b2b5360b36e47fb82e3f427a59de71fa1660426 WatchSource:0}: Error finding container fa18c494daecec9b3a2c505b8b2b5360b36e47fb82e3f427a59de71fa1660426: Status 404 returned error can't find the container with id fa18c494daecec9b3a2c505b8b2b5360b36e47fb82e3f427a59de71fa1660426
Sep 09 20:19:13.526703 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:13.526670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q" event={"ID":"f386a589-3e94-4453-a100-b49fc67952f1","Type":"ContainerStarted","Data":"fa18c494daecec9b3a2c505b8b2b5360b36e47fb82e3f427a59de71fa1660426"}
Sep 09 20:19:16.542027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:16.541990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q" event={"ID":"f386a589-3e94-4453-a100-b49fc67952f1","Type":"ContainerStarted","Data":"be2d556074cf575e94f8568e2d6a109bc96b4c437a1fae03224bb3978481fc5f"}
Sep 09 20:19:16.561365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:16.561309    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5fq7q" podStartSLOduration=2.09949568 podStartE2EDuration="4.561294453s" totalImagesPullingTime="2.461798773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:16.560051395 +0000 UTC m=+481.595522556" watchObservedRunningTime="2026-09-09 20:19:16.561294453 +0000 UTC m=+481.596765613"
Sep 09 20:19:32.053258 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.053219    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"]
Sep 09 20:19:32.057660 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.057641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.061217 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.061192    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 20:19:32.061217 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.061216    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 20:19:32.061408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.061390    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 20:19:32.061475 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.061467    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-h728d\""
Sep 09 20:19:32.066151 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.066128    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"]
Sep 09 20:19:32.109143 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.109076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8720a49b-11fe-48e3-b46b-34283524c366-cert\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.109316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.109155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhnx\" (UniqueName: \"kubernetes.io/projected/8720a49b-11fe-48e3-b46b-34283524c366-kube-api-access-kfhnx\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.109316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.109209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8720a49b-11fe-48e3-b46b-34283524c366-metrics-cert\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.109316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.109275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8720a49b-11fe-48e3-b46b-34283524c366-manager-config\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.210371 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.210334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8720a49b-11fe-48e3-b46b-34283524c366-cert\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.210541 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.210384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhnx\" (UniqueName: \"kubernetes.io/projected/8720a49b-11fe-48e3-b46b-34283524c366-kube-api-access-kfhnx\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.210541 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.210411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8720a49b-11fe-48e3-b46b-34283524c366-metrics-cert\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.210541 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.210460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8720a49b-11fe-48e3-b46b-34283524c366-manager-config\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.211070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.211047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8720a49b-11fe-48e3-b46b-34283524c366-manager-config\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.212973 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.212956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8720a49b-11fe-48e3-b46b-34283524c366-metrics-cert\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.213139 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.213117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8720a49b-11fe-48e3-b46b-34283524c366-cert\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.219712 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.219692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhnx\" (UniqueName: \"kubernetes.io/projected/8720a49b-11fe-48e3-b46b-34283524c366-kube-api-access-kfhnx\") pod \"lws-controller-manager-7f7677c75-wvxsr\" (UID: \"8720a49b-11fe-48e3-b46b-34283524c366\") " pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.367155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.367121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:32.637289 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:32.637263    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"]
Sep 09 20:19:32.639655 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:19:32.639626    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8720a49b_11fe_48e3_b46b_34283524c366.slice/crio-6ea58b512cb79d72a4b874562ba179365112ba3b7adf4debf67376dc2163adfe WatchSource:0}: Error finding container 6ea58b512cb79d72a4b874562ba179365112ba3b7adf4debf67376dc2163adfe: Status 404 returned error can't find the container with id 6ea58b512cb79d72a4b874562ba179365112ba3b7adf4debf67376dc2163adfe
Sep 09 20:19:33.618787 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:33.618747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr" event={"ID":"8720a49b-11fe-48e3-b46b-34283524c366","Type":"ContainerStarted","Data":"6ea58b512cb79d72a4b874562ba179365112ba3b7adf4debf67376dc2163adfe"}
Sep 09 20:19:36.632420 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:36.632382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr" event={"ID":"8720a49b-11fe-48e3-b46b-34283524c366","Type":"ContainerStarted","Data":"2e135f05a3b6a86829ce47fd7320bc568a9d3a7dea091728491621d22e29fc5f"}
Sep 09 20:19:36.632918 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:36.632476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:19:36.649134 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:36.649066    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr" podStartSLOduration=1.640754159 podStartE2EDuration="4.649054571s" totalImagesPullingTime="3.008300412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:36.647799516 +0000 UTC m=+501.683270676" watchObservedRunningTime="2026-09-09 20:19:36.649054571 +0000 UTC m=+501.684525732"
Sep 09 20:19:47.639140 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:19:47.639105    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f7677c75-wvxsr"
Sep 09 20:20:12.931232 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.931158    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"]
Sep 09 20:20:12.935789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.935766    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:12.939791 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.939766    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:20:12.939791 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.939765    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 20:20:12.939957 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.939813    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 20:20:12.940689 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.940672    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9mpkd\""
Sep 09 20:20:12.940776 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.940709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:20:12.949014 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.948992    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"]
Sep 09 20:20:12.958675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.958645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a612c1-9152-41a3-952a-5e85f2de5bb1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:12.958777 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.958712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16a612c1-9152-41a3-952a-5e85f2de5bb1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:12.958777 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:12.958768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwmz\" (UniqueName: \"kubernetes.io/projected/16a612c1-9152-41a3-952a-5e85f2de5bb1-kube-api-access-7xwmz\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.060054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.060021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a612c1-9152-41a3-952a-5e85f2de5bb1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.060252 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.060066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16a612c1-9152-41a3-952a-5e85f2de5bb1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.060252 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.060128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwmz\" (UniqueName: \"kubernetes.io/projected/16a612c1-9152-41a3-952a-5e85f2de5bb1-kube-api-access-7xwmz\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.060379 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:20:13.060255    2566 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 20:20:13.060379 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:20:13.060340    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a612c1-9152-41a3-952a-5e85f2de5bb1-plugin-serving-cert podName:16a612c1-9152-41a3-952a-5e85f2de5bb1 nodeName:}" failed. No retries permitted until 2026-09-09 20:20:13.560317091 +0000 UTC m=+538.595788244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/16a612c1-9152-41a3-952a-5e85f2de5bb1-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-kr2db" (UID: "16a612c1-9152-41a3-952a-5e85f2de5bb1") : secret "plugin-serving-cert" not found
Sep 09 20:20:13.060685 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.060663    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a612c1-9152-41a3-952a-5e85f2de5bb1-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.071936 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.071916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwmz\" (UniqueName: \"kubernetes.io/projected/16a612c1-9152-41a3-952a-5e85f2de5bb1-kube-api-access-7xwmz\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.564040 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.564006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16a612c1-9152-41a3-952a-5e85f2de5bb1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.566645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.566616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16a612c1-9152-41a3-952a-5e85f2de5bb1-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kr2db\" (UID: \"16a612c1-9152-41a3-952a-5e85f2de5bb1\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:13.845110 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:13.845027    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"
Sep 09 20:20:14.120659 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:14.120476    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db"]
Sep 09 20:20:14.123376 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:20:14.123353    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16a612c1_9152_41a3_952a_5e85f2de5bb1.slice/crio-1c922e60755b831d50506164c7f753115f29f231d0a779e6b15a69d541195593 WatchSource:0}: Error finding container 1c922e60755b831d50506164c7f753115f29f231d0a779e6b15a69d541195593: Status 404 returned error can't find the container with id 1c922e60755b831d50506164c7f753115f29f231d0a779e6b15a69d541195593
Sep 09 20:20:14.795225 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:14.795194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db" event={"ID":"16a612c1-9152-41a3-952a-5e85f2de5bb1","Type":"ContainerStarted","Data":"1c922e60755b831d50506164c7f753115f29f231d0a779e6b15a69d541195593"}
Sep 09 20:20:19.820663 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:19.820615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db" event={"ID":"16a612c1-9152-41a3-952a-5e85f2de5bb1","Type":"ContainerStarted","Data":"2823f41582712d1b5ff3c2485d2ba18fc95fd753f57a53d5581da0205ace130c"}
Sep 09 20:20:19.837760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:19.837706    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kr2db" podStartSLOduration=3.152465213 podStartE2EDuration="7.837690518s" totalImagesPullingTime="4.685225305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:19.836451144 +0000 UTC m=+544.871922297" watchObservedRunningTime="2026-09-09 20:20:19.837690518 +0000 UTC m=+544.873161678"
Sep 09 20:20:56.993351 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:56.993307    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fm84r"]
Sep 09 20:20:57.017317 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.017270    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fm84r"]
Sep 09 20:20:57.017504 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.017393    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r"
Sep 09 20:20:57.019860 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.019835    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4rg96\""
Sep 09 20:20:57.049057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.049009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2v9\" (UniqueName: \"kubernetes.io/projected/a4d4f176-a3d6-4740-8aa2-90849e5ca693-kube-api-access-7t2v9\") pod \"authorino-7bcfd8f9cd-fm84r\" (UID: \"a4d4f176-a3d6-4740-8aa2-90849e5ca693\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r"
Sep 09 20:20:57.150326 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.150283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2v9\" (UniqueName: \"kubernetes.io/projected/a4d4f176-a3d6-4740-8aa2-90849e5ca693-kube-api-access-7t2v9\") pod \"authorino-7bcfd8f9cd-fm84r\" (UID: \"a4d4f176-a3d6-4740-8aa2-90849e5ca693\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r"
Sep 09 20:20:57.158718 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.158689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2v9\" (UniqueName: \"kubernetes.io/projected/a4d4f176-a3d6-4740-8aa2-90849e5ca693-kube-api-access-7t2v9\") pod \"authorino-7bcfd8f9cd-fm84r\" (UID: \"a4d4f176-a3d6-4740-8aa2-90849e5ca693\") " pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r"
Sep 09 20:20:57.327700 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.327613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r"
Sep 09 20:20:57.399381 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.398723    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-fr4k2"]
Sep 09 20:20:57.402349 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.402319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fr4k2"
Sep 09 20:20:57.410027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.409782    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fr4k2"]
Sep 09 20:20:57.453399 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.453356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4t22\" (UniqueName: \"kubernetes.io/projected/45fcdf41-6159-476d-8c86-e9a046e958d7-kube-api-access-d4t22\") pod \"authorino-59c974bd48-fr4k2\" (UID: \"45fcdf41-6159-476d-8c86-e9a046e958d7\") " pod="kuadrant-system/authorino-59c974bd48-fr4k2"
Sep 09 20:20:57.554610 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.554571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4t22\" (UniqueName: \"kubernetes.io/projected/45fcdf41-6159-476d-8c86-e9a046e958d7-kube-api-access-d4t22\") pod \"authorino-59c974bd48-fr4k2\" (UID: \"45fcdf41-6159-476d-8c86-e9a046e958d7\") " pod="kuadrant-system/authorino-59c974bd48-fr4k2"
Sep 09 20:20:57.564167 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.564131    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4t22\" (UniqueName: \"kubernetes.io/projected/45fcdf41-6159-476d-8c86-e9a046e958d7-kube-api-access-d4t22\") pod \"authorino-59c974bd48-fr4k2\" (UID: \"45fcdf41-6159-476d-8c86-e9a046e958d7\") " pod="kuadrant-system/authorino-59c974bd48-fr4k2"
Sep 09 20:20:57.597809 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.597782    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fm84r"]
Sep 09 20:20:57.599986 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:20:57.599952    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d4f176_a3d6_4740_8aa2_90849e5ca693.slice/crio-ce6f62b7e3a7d3987674b1868684c0f7a3340d3e9e93e346a1a2e3dd3d65df58 WatchSource:0}: Error finding container ce6f62b7e3a7d3987674b1868684c0f7a3340d3e9e93e346a1a2e3dd3d65df58: Status 404 returned error can't find the container with id ce6f62b7e3a7d3987674b1868684c0f7a3340d3e9e93e346a1a2e3dd3d65df58
Sep 09 20:20:57.712550 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.712505    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fr4k2"
Sep 09 20:20:57.983554 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.983451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fr4k2"]
Sep 09 20:20:57.985808 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:20:57.985778    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45fcdf41_6159_476d_8c86_e9a046e958d7.slice/crio-e16b7130b0fa4cb8b95aad6142e999a51caf1e7bf6e259b4dab870be05703af4 WatchSource:0}: Error finding container e16b7130b0fa4cb8b95aad6142e999a51caf1e7bf6e259b4dab870be05703af4: Status 404 returned error can't find the container with id e16b7130b0fa4cb8b95aad6142e999a51caf1e7bf6e259b4dab870be05703af4
Sep 09 20:20:57.991511 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:57.991485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r" event={"ID":"a4d4f176-a3d6-4740-8aa2-90849e5ca693","Type":"ContainerStarted","Data":"ce6f62b7e3a7d3987674b1868684c0f7a3340d3e9e93e346a1a2e3dd3d65df58"}
Sep 09 20:20:58.999989 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:20:58.999943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fr4k2" event={"ID":"45fcdf41-6159-476d-8c86-e9a046e958d7","Type":"ContainerStarted","Data":"e16b7130b0fa4cb8b95aad6142e999a51caf1e7bf6e259b4dab870be05703af4"}
Sep 09 20:21:00.005268 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:00.005227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r" event={"ID":"a4d4f176-a3d6-4740-8aa2-90849e5ca693","Type":"ContainerStarted","Data":"6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f"}
Sep 09 20:21:00.006714 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:00.006689    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fr4k2" event={"ID":"45fcdf41-6159-476d-8c86-e9a046e958d7","Type":"ContainerStarted","Data":"2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a"}
Sep 09 20:21:00.019787 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:00.019743    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r" podStartSLOduration=2.217997382 podStartE2EDuration="4.019729207s" totalImagesPullingTime="1.801731825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:00.019053876 +0000 UTC m=+585.054525036" watchObservedRunningTime="2026-09-09 20:21:00.019729207 +0000 UTC m=+585.055200366"
Sep 09 20:21:00.033412 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:00.033363    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-fr4k2" podStartSLOduration=1.613965999 podStartE2EDuration="3.033345294s" totalImagesPullingTime="1.419379295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:00.032837466 +0000 UTC m=+585.068308627" watchObservedRunningTime="2026-09-09 20:21:00.033345294 +0000 UTC m=+585.068816455"
Sep 09 20:21:00.058661 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:00.058631    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fm84r"]
Sep 09 20:21:02.014934 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:02.014897    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r" podUID="a4d4f176-a3d6-4740-8aa2-90849e5ca693" containerName="authorino" containerID="cri-o://6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f" gracePeriod=30
Sep 09 20:21:02.263523 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:02.263494    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r"
Sep 09 20:21:02.297451 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:02.297372    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t2v9\" (UniqueName: \"kubernetes.io/projected/a4d4f176-a3d6-4740-8aa2-90849e5ca693-kube-api-access-7t2v9\") pod \"a4d4f176-a3d6-4740-8aa2-90849e5ca693\" (UID: \"a4d4f176-a3d6-4740-8aa2-90849e5ca693\") "
Sep 09 20:21:02.299608 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:02.299578    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d4f176-a3d6-4740-8aa2-90849e5ca693-kube-api-access-7t2v9" (OuterVolumeSpecName: "kube-api-access-7t2v9") pod "a4d4f176-a3d6-4740-8aa2-90849e5ca693" (UID: "a4d4f176-a3d6-4740-8aa2-90849e5ca693"). InnerVolumeSpecName "kube-api-access-7t2v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:02.398306 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:02.398261    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2v9\" (UniqueName: \"kubernetes.io/projected/a4d4f176-a3d6-4740-8aa2-90849e5ca693-kube-api-access-7t2v9\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:21:03.020902 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.020860    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4d4f176-a3d6-4740-8aa2-90849e5ca693" containerID="6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f" exitCode=0
Sep 09 20:21:03.020902 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.020906    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r"
Sep 09 20:21:03.021467 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.020915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r" event={"ID":"a4d4f176-a3d6-4740-8aa2-90849e5ca693","Type":"ContainerDied","Data":"6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f"}
Sep 09 20:21:03.021467 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.020958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-fm84r" event={"ID":"a4d4f176-a3d6-4740-8aa2-90849e5ca693","Type":"ContainerDied","Data":"ce6f62b7e3a7d3987674b1868684c0f7a3340d3e9e93e346a1a2e3dd3d65df58"}
Sep 09 20:21:03.021467 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.020978    2566 scope.go:117] "RemoveContainer" containerID="6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f"
Sep 09 20:21:03.030154 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.030073    2566 scope.go:117] "RemoveContainer" containerID="6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f"
Sep 09 20:21:03.030443 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:21:03.030425    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f\": container with ID starting with 6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f not found: ID does not exist" containerID="6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f"
Sep 09 20:21:03.030496 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.030452    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f"} err="failed to get container status \"6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f\": rpc error: code = NotFound desc = could not find container \"6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f\": container with ID starting with 6b564ccc6779f8eb601f5cceb99ec0fa4afa33bcfea590fb5f7ba0845173e21f not found: ID does not exist"
Sep 09 20:21:03.042394 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.042371    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fm84r"]
Sep 09 20:21:03.048137 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.048113    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-fm84r"]
Sep 09 20:21:03.490433 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:03.490400    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d4f176-a3d6-4740-8aa2-90849e5ca693" path="/var/lib/kubelet/pods/a4d4f176-a3d6-4740-8aa2-90849e5ca693/volumes"
Sep 09 20:21:19.519048 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.519009    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-z6mfr"]
Sep 09 20:21:19.519481 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.519352    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d4f176-a3d6-4740-8aa2-90849e5ca693" containerName="authorino"
Sep 09 20:21:19.519481 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.519365    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d4f176-a3d6-4740-8aa2-90849e5ca693" containerName="authorino"
Sep 09 20:21:19.519481 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.519414    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d4f176-a3d6-4740-8aa2-90849e5ca693" containerName="authorino"
Sep 09 20:21:19.523544 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.523523    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:19.525961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.525928    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 20:21:19.529935 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.529908    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-z6mfr"]
Sep 09 20:21:19.645251 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.645211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tsh\" (UniqueName: \"kubernetes.io/projected/63b8a9e1-d5c9-4ec9-b774-ae9607616c88-kube-api-access-j4tsh\") pod \"authorino-6d9898bd99-z6mfr\" (UID: \"63b8a9e1-d5c9-4ec9-b774-ae9607616c88\") " pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:19.645251 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.645254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63b8a9e1-d5c9-4ec9-b774-ae9607616c88-tls-cert\") pod \"authorino-6d9898bd99-z6mfr\" (UID: \"63b8a9e1-d5c9-4ec9-b774-ae9607616c88\") " pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:19.746509 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.746475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tsh\" (UniqueName: \"kubernetes.io/projected/63b8a9e1-d5c9-4ec9-b774-ae9607616c88-kube-api-access-j4tsh\") pod \"authorino-6d9898bd99-z6mfr\" (UID: \"63b8a9e1-d5c9-4ec9-b774-ae9607616c88\") " pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:19.746616 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.746514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63b8a9e1-d5c9-4ec9-b774-ae9607616c88-tls-cert\") pod \"authorino-6d9898bd99-z6mfr\" (UID: \"63b8a9e1-d5c9-4ec9-b774-ae9607616c88\") " pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:19.749102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.749055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63b8a9e1-d5c9-4ec9-b774-ae9607616c88-tls-cert\") pod \"authorino-6d9898bd99-z6mfr\" (UID: \"63b8a9e1-d5c9-4ec9-b774-ae9607616c88\") " pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:19.756589 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.756562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tsh\" (UniqueName: \"kubernetes.io/projected/63b8a9e1-d5c9-4ec9-b774-ae9607616c88-kube-api-access-j4tsh\") pod \"authorino-6d9898bd99-z6mfr\" (UID: \"63b8a9e1-d5c9-4ec9-b774-ae9607616c88\") " pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:19.834193 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:19.834108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-z6mfr"
Sep 09 20:21:20.105030 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:20.105006    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-z6mfr"]
Sep 09 20:21:20.107334 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:21:20.107299    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63b8a9e1_d5c9_4ec9_b774_ae9607616c88.slice/crio-74f6040f531208f12f6c0717558eb5dca6276a036f8e909e5ab9060149090548 WatchSource:0}: Error finding container 74f6040f531208f12f6c0717558eb5dca6276a036f8e909e5ab9060149090548: Status 404 returned error can't find the container with id 74f6040f531208f12f6c0717558eb5dca6276a036f8e909e5ab9060149090548
Sep 09 20:21:21.097715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.097625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-z6mfr" event={"ID":"63b8a9e1-d5c9-4ec9-b774-ae9607616c88","Type":"ContainerStarted","Data":"9134c0e95303e40097b8a07158dc053e7ea4106bb837e7e8ae11cfa99c944716"}
Sep 09 20:21:21.097715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.097668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-z6mfr" event={"ID":"63b8a9e1-d5c9-4ec9-b774-ae9607616c88","Type":"ContainerStarted","Data":"74f6040f531208f12f6c0717558eb5dca6276a036f8e909e5ab9060149090548"}
Sep 09 20:21:21.116102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.116033    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-z6mfr" podStartSLOduration=1.4110053869999999 podStartE2EDuration="2.116017844s" totalImagesPullingTime="705.012457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:21.114531184 +0000 UTC m=+606.150002343" watchObservedRunningTime="2026-09-09 20:21:21.116017844 +0000 UTC m=+606.151489003"
Sep 09 20:21:21.141556 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.141522    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fr4k2"]
Sep 09 20:21:21.141760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.141734    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-fr4k2" podUID="45fcdf41-6159-476d-8c86-e9a046e958d7" containerName="authorino" containerID="cri-o://2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a" gracePeriod=30
Sep 09 20:21:21.377073 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.377047    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fr4k2"
Sep 09 20:21:21.461060 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.461028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4t22\" (UniqueName: \"kubernetes.io/projected/45fcdf41-6159-476d-8c86-e9a046e958d7-kube-api-access-d4t22\") pod \"45fcdf41-6159-476d-8c86-e9a046e958d7\" (UID: \"45fcdf41-6159-476d-8c86-e9a046e958d7\") "
Sep 09 20:21:21.463220 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.463196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fcdf41-6159-476d-8c86-e9a046e958d7-kube-api-access-d4t22" (OuterVolumeSpecName: "kube-api-access-d4t22") pod "45fcdf41-6159-476d-8c86-e9a046e958d7" (UID: "45fcdf41-6159-476d-8c86-e9a046e958d7"). InnerVolumeSpecName "kube-api-access-d4t22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:21.562523 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:21.562490    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4t22\" (UniqueName: \"kubernetes.io/projected/45fcdf41-6159-476d-8c86-e9a046e958d7-kube-api-access-d4t22\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:21:22.103123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.103027    2566 generic.go:356] "Generic (PLEG): container finished" podID="45fcdf41-6159-476d-8c86-e9a046e958d7" containerID="2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a" exitCode=0
Sep 09 20:21:22.103123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.103069    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-fr4k2"
Sep 09 20:21:22.103585 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.103135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fr4k2" event={"ID":"45fcdf41-6159-476d-8c86-e9a046e958d7","Type":"ContainerDied","Data":"2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a"}
Sep 09 20:21:22.103585 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.103183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-fr4k2" event={"ID":"45fcdf41-6159-476d-8c86-e9a046e958d7","Type":"ContainerDied","Data":"e16b7130b0fa4cb8b95aad6142e999a51caf1e7bf6e259b4dab870be05703af4"}
Sep 09 20:21:22.103585 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.103206    2566 scope.go:117] "RemoveContainer" containerID="2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a"
Sep 09 20:21:22.111645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.111629    2566 scope.go:117] "RemoveContainer" containerID="2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a"
Sep 09 20:21:22.111931 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:21:22.111911    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a\": container with ID starting with 2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a not found: ID does not exist" containerID="2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a"
Sep 09 20:21:22.111990 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.111938    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a"} err="failed to get container status \"2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a\": rpc error: code = NotFound desc = could not find container \"2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a\": container with ID starting with 2c90480a616ea3efe7f41d37c44b1e571f87e03cdf33bc9e7aacf96d3677ee8a not found: ID does not exist"
Sep 09 20:21:22.121064 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.121040    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fr4k2"]
Sep 09 20:21:22.125959 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:22.125937    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-fr4k2"]
Sep 09 20:21:23.490952 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:23.490917    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45fcdf41-6159-476d-8c86-e9a046e958d7" path="/var/lib/kubelet/pods/45fcdf41-6159-476d-8c86-e9a046e958d7/volumes"
Sep 09 20:21:50.592039 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.591965    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-75f754565b-jzxdx"]
Sep 09 20:21:50.592550 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.592378    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fcdf41-6159-476d-8c86-e9a046e958d7" containerName="authorino"
Sep 09 20:21:50.592550 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.592396    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fcdf41-6159-476d-8c86-e9a046e958d7" containerName="authorino"
Sep 09 20:21:50.592550 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.592482    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fcdf41-6159-476d-8c86-e9a046e958d7" containerName="authorino"
Sep 09 20:21:50.594534 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.594510    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:50.597215 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.597191    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:21:50.597372 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.597345    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-zmc5v\""
Sep 09 20:21:50.597372 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.597361    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:21:50.597498 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.597405    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:21:50.602821 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.602802    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-75f754565b-jzxdx"]
Sep 09 20:21:50.609562 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.609540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/d3e47588-3ef6-4fcc-b2ed-813cd9001713-kube-api-access-49h72\") pod \"llmisvc-controller-manager-75f754565b-jzxdx\" (UID: \"d3e47588-3ef6-4fcc-b2ed-813cd9001713\") " pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:50.609651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.609575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3e47588-3ef6-4fcc-b2ed-813cd9001713-cert\") pod \"llmisvc-controller-manager-75f754565b-jzxdx\" (UID: \"d3e47588-3ef6-4fcc-b2ed-813cd9001713\") " pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:50.710376 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.710345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/d3e47588-3ef6-4fcc-b2ed-813cd9001713-kube-api-access-49h72\") pod \"llmisvc-controller-manager-75f754565b-jzxdx\" (UID: \"d3e47588-3ef6-4fcc-b2ed-813cd9001713\") " pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:50.710515 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.710382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3e47588-3ef6-4fcc-b2ed-813cd9001713-cert\") pod \"llmisvc-controller-manager-75f754565b-jzxdx\" (UID: \"d3e47588-3ef6-4fcc-b2ed-813cd9001713\") " pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:50.712963 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.712931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3e47588-3ef6-4fcc-b2ed-813cd9001713-cert\") pod \"llmisvc-controller-manager-75f754565b-jzxdx\" (UID: \"d3e47588-3ef6-4fcc-b2ed-813cd9001713\") " pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:50.718794 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.718769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49h72\" (UniqueName: \"kubernetes.io/projected/d3e47588-3ef6-4fcc-b2ed-813cd9001713-kube-api-access-49h72\") pod \"llmisvc-controller-manager-75f754565b-jzxdx\" (UID: \"d3e47588-3ef6-4fcc-b2ed-813cd9001713\") " pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:50.905963 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:50.905918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:51.181191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:51.181159    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-75f754565b-jzxdx"]
Sep 09 20:21:51.183625 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:21:51.183595    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd3e47588_3ef6_4fcc_b2ed_813cd9001713.slice/crio-b0dfa4e358e622177fcdd2c2cb07a255606728fe918e9e803aed6811eb7bed79 WatchSource:0}: Error finding container b0dfa4e358e622177fcdd2c2cb07a255606728fe918e9e803aed6811eb7bed79: Status 404 returned error can't find the container with id b0dfa4e358e622177fcdd2c2cb07a255606728fe918e9e803aed6811eb7bed79
Sep 09 20:21:51.233420 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:51.233388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx" event={"ID":"d3e47588-3ef6-4fcc-b2ed-813cd9001713","Type":"ContainerStarted","Data":"b0dfa4e358e622177fcdd2c2cb07a255606728fe918e9e803aed6811eb7bed79"}
Sep 09 20:21:54.250954 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:54.250854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx" event={"ID":"d3e47588-3ef6-4fcc-b2ed-813cd9001713","Type":"ContainerStarted","Data":"e98b0df8ceac90d7110e7a97ad27ee269593dc36c2c3105333ccffc2bcc92386"}
Sep 09 20:21:54.251361 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:54.250981    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:21:54.270896 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:21:54.270846    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx" podStartSLOduration=1.48170589 podStartE2EDuration="4.270829367s" totalImagesPullingTime="2.789123477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:54.269966481 +0000 UTC m=+639.305437642" watchObservedRunningTime="2026-09-09 20:21:54.270829367 +0000 UTC m=+639.306300526"
Sep 09 20:22:25.257949 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:22:25.257913    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-75f754565b-jzxdx"
Sep 09 20:23:05.920728 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:05.920694    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-svwj5"]
Sep 09 20:23:05.924106 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:05.924073    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:05.926421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:05.926398    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:23:05.926540 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:05.926482    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-clprd\""
Sep 09 20:23:05.932775 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:05.932752    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-svwj5"]
Sep 09 20:23:06.021399 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.021366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp54h\" (UniqueName: \"kubernetes.io/projected/310bdb08-7715-4a7f-baea-c03fdcc6795c-kube-api-access-rp54h\") pod \"model-serving-api-7b954ffc95-svwj5\" (UID: \"310bdb08-7715-4a7f-baea-c03fdcc6795c\") " pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:06.021535 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.021407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310bdb08-7715-4a7f-baea-c03fdcc6795c-tls-certs\") pod \"model-serving-api-7b954ffc95-svwj5\" (UID: \"310bdb08-7715-4a7f-baea-c03fdcc6795c\") " pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:06.122635 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.122603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp54h\" (UniqueName: \"kubernetes.io/projected/310bdb08-7715-4a7f-baea-c03fdcc6795c-kube-api-access-rp54h\") pod \"model-serving-api-7b954ffc95-svwj5\" (UID: \"310bdb08-7715-4a7f-baea-c03fdcc6795c\") " pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:06.122790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.122643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310bdb08-7715-4a7f-baea-c03fdcc6795c-tls-certs\") pod \"model-serving-api-7b954ffc95-svwj5\" (UID: \"310bdb08-7715-4a7f-baea-c03fdcc6795c\") " pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:06.125330 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.125306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/310bdb08-7715-4a7f-baea-c03fdcc6795c-tls-certs\") pod \"model-serving-api-7b954ffc95-svwj5\" (UID: \"310bdb08-7715-4a7f-baea-c03fdcc6795c\") " pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:06.134340 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.134313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp54h\" (UniqueName: \"kubernetes.io/projected/310bdb08-7715-4a7f-baea-c03fdcc6795c-kube-api-access-rp54h\") pod \"model-serving-api-7b954ffc95-svwj5\" (UID: \"310bdb08-7715-4a7f-baea-c03fdcc6795c\") " pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:06.235457 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.235398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:06.506226 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.506204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-svwj5"]
Sep 09 20:23:06.508806 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:23:06.508769    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310bdb08_7715_4a7f_baea_c03fdcc6795c.slice/crio-20addb6293851657e388acd8a037d0a66b51950d120b1d0d806abaeb0d1aa386 WatchSource:0}: Error finding container 20addb6293851657e388acd8a037d0a66b51950d120b1d0d806abaeb0d1aa386: Status 404 returned error can't find the container with id 20addb6293851657e388acd8a037d0a66b51950d120b1d0d806abaeb0d1aa386
Sep 09 20:23:06.510837 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.510816    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:23:06.559870 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:06.559817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-svwj5" event={"ID":"310bdb08-7715-4a7f-baea-c03fdcc6795c","Type":"ContainerStarted","Data":"20addb6293851657e388acd8a037d0a66b51950d120b1d0d806abaeb0d1aa386"}
Sep 09 20:23:08.571501 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:08.571465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-svwj5" event={"ID":"310bdb08-7715-4a7f-baea-c03fdcc6795c","Type":"ContainerStarted","Data":"e4fac8cef387d922b772fa03cee0ee46f9550d65200e0f80d5a5e02b77468f65"}
Sep 09 20:23:08.571898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:08.571589    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:08.590645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:08.590598    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-svwj5" podStartSLOduration=2.204368271 podStartE2EDuration="3.590585219s" totalImagesPullingTime="1.386216948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:08.589474755 +0000 UTC m=+713.624945914" watchObservedRunningTime="2026-09-09 20:23:08.590585219 +0000 UTC m=+713.626056378"
Sep 09 20:23:19.580827 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:19.580796    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-svwj5"
Sep 09 20:23:21.605651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.605617    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-wrzkv"]
Sep 09 20:23:21.609066 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.609051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wrzkv"
Sep 09 20:23:21.611382 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.611354    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:23:21.611525 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.611457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:23:21.611577 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.611529    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-w62ns\""
Sep 09 20:23:21.619481 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.619457    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-wrzkv"]
Sep 09 20:23:21.746487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.746452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvcx\" (UniqueName: \"kubernetes.io/projected/9e9e9ccc-16da-46ab-8bf3-baae89a73785-kube-api-access-5bvcx\") pod \"s3-init-wrzkv\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") " pod="kserve/s3-init-wrzkv"
Sep 09 20:23:21.746672 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.746493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9e9ccc-16da-46ab-8bf3-baae89a73785-model-cache\") pod \"s3-init-wrzkv\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") " pod="kserve/s3-init-wrzkv"
Sep 09 20:23:21.847901 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.847874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvcx\" (UniqueName: \"kubernetes.io/projected/9e9e9ccc-16da-46ab-8bf3-baae89a73785-kube-api-access-5bvcx\") pod \"s3-init-wrzkv\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") " pod="kserve/s3-init-wrzkv"
Sep 09 20:23:21.848022 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.847908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9e9ccc-16da-46ab-8bf3-baae89a73785-model-cache\") pod \"s3-init-wrzkv\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") " pod="kserve/s3-init-wrzkv"
Sep 09 20:23:21.848291 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.848276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9e9ccc-16da-46ab-8bf3-baae89a73785-model-cache\") pod \"s3-init-wrzkv\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") " pod="kserve/s3-init-wrzkv"
Sep 09 20:23:21.857049 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.856992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvcx\" (UniqueName: \"kubernetes.io/projected/9e9e9ccc-16da-46ab-8bf3-baae89a73785-kube-api-access-5bvcx\") pod \"s3-init-wrzkv\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") " pod="kserve/s3-init-wrzkv"
Sep 09 20:23:21.918588 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:21.918565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wrzkv"
Sep 09 20:23:22.188679 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:22.188656    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-wrzkv"]
Sep 09 20:23:22.190817 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:23:22.190789    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e9e9ccc_16da_46ab_8bf3_baae89a73785.slice/crio-2e818aaf1474414e5b67efdb4ff484e578920f21b9ececb1283c5f01fb20ccdf WatchSource:0}: Error finding container 2e818aaf1474414e5b67efdb4ff484e578920f21b9ececb1283c5f01fb20ccdf: Status 404 returned error can't find the container with id 2e818aaf1474414e5b67efdb4ff484e578920f21b9ececb1283c5f01fb20ccdf
Sep 09 20:23:22.641443 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:22.641401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wrzkv" event={"ID":"9e9e9ccc-16da-46ab-8bf3-baae89a73785","Type":"ContainerStarted","Data":"2e818aaf1474414e5b67efdb4ff484e578920f21b9ececb1283c5f01fb20ccdf"}
Sep 09 20:23:26.668326 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:26.668282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wrzkv" event={"ID":"9e9e9ccc-16da-46ab-8bf3-baae89a73785","Type":"ContainerStarted","Data":"819cde3925447619317d9d6b645106e55d26aaa35ee5f0ea0c5b4557d1d5550f"}
Sep 09 20:23:27.674206 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:27.674169    2566 generic.go:356] "Generic (PLEG): container finished" podID="9e9e9ccc-16da-46ab-8bf3-baae89a73785" containerID="819cde3925447619317d9d6b645106e55d26aaa35ee5f0ea0c5b4557d1d5550f" exitCode=0
Sep 09 20:23:27.674544 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:27.674259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wrzkv" event={"ID":"9e9e9ccc-16da-46ab-8bf3-baae89a73785","Type":"ContainerDied","Data":"819cde3925447619317d9d6b645106e55d26aaa35ee5f0ea0c5b4557d1d5550f"}
Sep 09 20:23:32.702354 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:32.702317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wrzkv" event={"ID":"9e9e9ccc-16da-46ab-8bf3-baae89a73785","Type":"ContainerStarted","Data":"8c2e1b3c782f4ab6031c79796692df71f42ff4e3bcfdbc97bc976993553eaf49"}
Sep 09 20:23:32.720941 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:23:32.720889    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-wrzkv" podStartSLOduration=2.230718003 podStartE2EDuration="11.720875393s" totalImagesPullingTime="7.725276183s" totalInitContainerRuntime="1.764881207s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:32.718241959 +0000 UTC m=+737.753713117" watchObservedRunningTime="2026-09-09 20:23:32.720875393 +0000 UTC m=+737.756346551"
Sep 09 20:24:36.988228 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:36.988189    2566 generic.go:356] "Generic (PLEG): container finished" podID="9e9e9ccc-16da-46ab-8bf3-baae89a73785" containerID="8c2e1b3c782f4ab6031c79796692df71f42ff4e3bcfdbc97bc976993553eaf49" exitCode=0
Sep 09 20:24:36.988675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:36.988248    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wrzkv" event={"ID":"9e9e9ccc-16da-46ab-8bf3-baae89a73785","Type":"ContainerDied","Data":"8c2e1b3c782f4ab6031c79796692df71f42ff4e3bcfdbc97bc976993553eaf49"}
Sep 09 20:24:38.131680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:38.131657    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wrzkv"
Sep 09 20:24:38.258531 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:38.258448    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9e9ccc-16da-46ab-8bf3-baae89a73785-model-cache\") pod \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") "
Sep 09 20:24:38.258531 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:38.258508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bvcx\" (UniqueName: \"kubernetes.io/projected/9e9e9ccc-16da-46ab-8bf3-baae89a73785-kube-api-access-5bvcx\") pod \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\" (UID: \"9e9e9ccc-16da-46ab-8bf3-baae89a73785\") "
Sep 09 20:24:38.260871 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:38.260845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9e9ccc-16da-46ab-8bf3-baae89a73785-kube-api-access-5bvcx" (OuterVolumeSpecName: "kube-api-access-5bvcx") pod "9e9e9ccc-16da-46ab-8bf3-baae89a73785" (UID: "9e9e9ccc-16da-46ab-8bf3-baae89a73785"). InnerVolumeSpecName "kube-api-access-5bvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:38.277219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:38.277187    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9e9ccc-16da-46ab-8bf3-baae89a73785-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e9e9ccc-16da-46ab-8bf3-baae89a73785" (UID: "9e9e9ccc-16da-46ab-8bf3-baae89a73785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:38.359626 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:38.359599    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e9e9ccc-16da-46ab-8bf3-baae89a73785-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:24:38.359626 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:38.359625    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bvcx\" (UniqueName: \"kubernetes.io/projected/9e9e9ccc-16da-46ab-8bf3-baae89a73785-kube-api-access-5bvcx\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:24:39.004895 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:39.004862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-wrzkv" event={"ID":"9e9e9ccc-16da-46ab-8bf3-baae89a73785","Type":"ContainerDied","Data":"2e818aaf1474414e5b67efdb4ff484e578920f21b9ececb1283c5f01fb20ccdf"}
Sep 09 20:24:39.004895 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:39.004898    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e818aaf1474414e5b67efdb4ff484e578920f21b9ececb1283c5f01fb20ccdf"
Sep 09 20:24:39.005111 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:24:39.004907    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-wrzkv"
Sep 09 20:25:00.856388 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.856351    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"]
Sep 09 20:25:00.856790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.856774    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9e9ccc-16da-46ab-8bf3-baae89a73785" containerName="download-hf-model"
Sep 09 20:25:00.856836 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.856793    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9e9ccc-16da-46ab-8bf3-baae89a73785" containerName="download-hf-model"
Sep 09 20:25:00.856836 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.856810    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9e9ccc-16da-46ab-8bf3-baae89a73785" containerName="s3-init"
Sep 09 20:25:00.856836 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.856815    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9e9ccc-16da-46ab-8bf3-baae89a73785" containerName="s3-init"
Sep 09 20:25:00.856931 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.856873    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9e9ccc-16da-46ab-8bf3-baae89a73785" containerName="s3-init"
Sep 09 20:25:00.859981 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.859961    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:00.862372 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.862347    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 09 20:25:00.862505 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.862448    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:25:00.862571 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.862516    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 09 20:25:00.862646 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.862628    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-cblnh\""
Sep 09 20:25:00.862708 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.862693    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 09 20:25:00.862766 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.862754    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 09 20:25:00.870009 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:00.869983    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"]
Sep 09 20:25:01.056215 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.056397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.056397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.056397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.056397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkjf\" (UniqueName: \"kubernetes.io/projected/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kube-api-access-jmkjf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.056613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.056613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.056613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.056543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157467 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157467 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkjf\" (UniqueName: \"kubernetes.io/projected/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kube-api-access-jmkjf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.157742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.158048 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.158048 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.157924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.158216 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.158196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.158398 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.158375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.158475 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.158375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.160118 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.160071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.160245 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.160178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.169037 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.169009    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkjf\" (UniqueName: \"kubernetes.io/projected/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kube-api-access-jmkjf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.170781 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.170751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:01.457075 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:01.457048    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"]
Sep 09 20:25:01.459654 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:25:01.459623    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4b1e7f_9cb8_42ae_ad0a_e85e4ab37bd5.slice/crio-c8a4f47323efc907efa196e5e3c4eba4d0ad1689151b8760559eedee5b0580b5 WatchSource:0}: Error finding container c8a4f47323efc907efa196e5e3c4eba4d0ad1689151b8760559eedee5b0580b5: Status 404 returned error can't find the container with id c8a4f47323efc907efa196e5e3c4eba4d0ad1689151b8760559eedee5b0580b5
Sep 09 20:25:02.108490 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:02.108452    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" event={"ID":"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5","Type":"ContainerStarted","Data":"1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b"}
Sep 09 20:25:02.108910 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:02.108504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" event={"ID":"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5","Type":"ContainerStarted","Data":"c8a4f47323efc907efa196e5e3c4eba4d0ad1689151b8760559eedee5b0580b5"}
Sep 09 20:25:04.120680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:04.120646    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerID="1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b" exitCode=0
Sep 09 20:25:04.121170 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:04.120728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" event={"ID":"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5","Type":"ContainerDied","Data":"1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b"}
Sep 09 20:25:35.297534 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:35.297499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" event={"ID":"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5","Type":"ContainerStarted","Data":"4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e"}
Sep 09 20:25:35.321474 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:35.321410    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podStartSLOduration=3.026206615 podStartE2EDuration="35.321392709s" totalImagesPullingTime="30.085708575s" totalInitContainerRuntime="2.209477519s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:35.318601464 +0000 UTC m=+860.354072625" watchObservedRunningTime="2026-09-09 20:25:35.321392709 +0000 UTC m=+860.356863890"
Sep 09 20:25:36.171760 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:36.171717    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:41.171182 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:41.171121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:25:41.172757 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:41.172720    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:25:51.172199 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:25:51.172148    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:26:01.171649 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:26:01.171606    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:26:11.171904 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:26:11.171811    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:26:21.171604 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:26:21.171519    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:26:31.172018 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:26:31.171968    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:26:41.171487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:26:41.171432    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:26:51.172069 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:26:51.172022    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:27:01.171637 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:27:01.171587    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:27:11.171698 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:27:11.171648    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:27:21.171732 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:27:21.171684    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:27:31.181432 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:27:31.181388    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:27:31.189939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:27:31.189907    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:27:53.858156 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:27:53.858051    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"]
Sep 09 20:28:08.883000 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:08.882954    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" containerID="cri-o://4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e" gracePeriod=45
Sep 09 20:28:09.182603 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.182513    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 09 20:28:09.626791 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.626767    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:28:09.743938 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.743912    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tmp-dir\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.743946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-cabundle-cert\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.743980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tls-certs\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-model-cache\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744039    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kserve-provision-location\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmkjf\" (UniqueName: \"kubernetes.io/projected/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kube-api-access-jmkjf\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744123 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744117    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-home\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744434 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-dshm\") pod \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\" (UID: \"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5\") "
Sep 09 20:28:09.744434 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744394    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:09.744528 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:09.744969 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.744945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-home" (OuterVolumeSpecName: "home") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:09.746799 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.746775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:09.746920 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.746889    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-dshm" (OuterVolumeSpecName: "dshm") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:09.746920 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.746899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kube-api-access-jmkjf" (OuterVolumeSpecName: "kube-api-access-jmkjf") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "kube-api-access-jmkjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:09.757383 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.757360    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:09.764294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.764270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" (UID: "bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:09.845640 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845610    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.845640 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845640    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.845640 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845648    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.845854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845656    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.845854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845665    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.845854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845673    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmkjf\" (UniqueName: \"kubernetes.io/projected/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-kube-api-access-jmkjf\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.845854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845680    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.845854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:09.845688    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:28:10.077525 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.077443    2566 generic.go:356] "Generic (PLEG): container finished" podID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerID="4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e" exitCode=0
Sep 09 20:28:10.077525 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.077494    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"
Sep 09 20:28:10.077978 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.077525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" event={"ID":"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5","Type":"ContainerDied","Data":"4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e"}
Sep 09 20:28:10.077978 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.077576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm" event={"ID":"bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5","Type":"ContainerDied","Data":"c8a4f47323efc907efa196e5e3c4eba4d0ad1689151b8760559eedee5b0580b5"}
Sep 09 20:28:10.077978 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.077599    2566 scope.go:117] "RemoveContainer" containerID="4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e"
Sep 09 20:28:10.088690 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.088673    2566 scope.go:117] "RemoveContainer" containerID="1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b"
Sep 09 20:28:10.096455 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.096436    2566 scope.go:117] "RemoveContainer" containerID="4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e"
Sep 09 20:28:10.096704 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:28:10.096685    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e\": container with ID starting with 4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e not found: ID does not exist" containerID="4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e"
Sep 09 20:28:10.096753 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.096712    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e"} err="failed to get container status \"4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e\": rpc error: code = NotFound desc = could not find container \"4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e\": container with ID starting with 4e53760c592fabde994dd4206aa506dcd976570654aa09b1513461486d43144e not found: ID does not exist"
Sep 09 20:28:10.096753 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.096728    2566 scope.go:117] "RemoveContainer" containerID="1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b"
Sep 09 20:28:10.096965 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:28:10.096949    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b\": container with ID starting with 1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b not found: ID does not exist" containerID="1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b"
Sep 09 20:28:10.097020 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.096974    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b"} err="failed to get container status \"1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b\": rpc error: code = NotFound desc = could not find container \"1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b\": container with ID starting with 1b41c7840095b4b2f16f597f02e56ffbcb777ca89e742bd05800fec890d6406b not found: ID does not exist"
Sep 09 20:28:10.103779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.103757    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"]
Sep 09 20:28:10.107602 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:10.107583    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c8cb9f4cckxckm"]
Sep 09 20:28:11.491102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:11.491056    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" path="/var/lib/kubelet/pods/bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5/volumes"
Sep 09 20:28:35.612562 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.612523    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 20:28:35.613149 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.612967    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="storage-initializer"
Sep 09 20:28:35.613149 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.612982    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="storage-initializer"
Sep 09 20:28:35.613149 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.613007    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main"
Sep 09 20:28:35.613149 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.613014    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main"
Sep 09 20:28:35.613149 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.613102    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4b1e7f-9cb8-42ae-ad0a-e85e4ab37bd5" containerName="main"
Sep 09 20:28:35.617496 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.617474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.620521 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.620493    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:28:35.621597 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.621579    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-qwdzz\""
Sep 09 20:28:35.621739 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.621603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 09 20:28:35.621811 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.621603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 09 20:28:35.621811 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.621622    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 09 20:28:35.621811 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.621622    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-ktjmc\""
Sep 09 20:28:35.621811 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.621674    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 09 20:28:35.633952 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.633876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 20:28:35.650719 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.650684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.650881 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.650745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k9m\" (UniqueName: \"kubernetes.io/projected/0bbff78c-22db-4ea3-8ba0-753c3046e692-kube-api-access-54k9m\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.650881 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.650792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.650881 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.650851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.651004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.650899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbff78c-22db-4ea3-8ba0-753c3046e692-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.651004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.650935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.651004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.650981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbff78c-22db-4ea3-8ba0-753c3046e692-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.651146 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.651011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751713 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbff78c-22db-4ea3-8ba0-753c3046e692-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751713 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751713 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbff78c-22db-4ea3-8ba0-753c3046e692-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751713 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751713 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751957 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.751957 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.752054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751962    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.752054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.751993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k9m\" (UniqueName: \"kubernetes.io/projected/0bbff78c-22db-4ea3-8ba0-753c3046e692-kube-api-access-54k9m\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.752187 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.752049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.752187 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.752165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbff78c-22db-4ea3-8ba0-753c3046e692-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.753971 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.753945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.754219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.754198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbff78c-22db-4ea3-8ba0-753c3046e692-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.760914 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.760889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k9m\" (UniqueName: \"kubernetes.io/projected/0bbff78c-22db-4ea3-8ba0-753c3046e692-kube-api-access-54k9m\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:35.935078 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:35.934984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:36.216966 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:36.216941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 20:28:36.220497 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:28:36.220465    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbff78c_22db_4ea3_8ba0_753c3046e692.slice/crio-7380c2a43fbfef684c4480649fd76bcf36946026070656a3456b94903ac578e3 WatchSource:0}: Error finding container 7380c2a43fbfef684c4480649fd76bcf36946026070656a3456b94903ac578e3: Status 404 returned error can't find the container with id 7380c2a43fbfef684c4480649fd76bcf36946026070656a3456b94903ac578e3
Sep 09 20:28:37.209103 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:37.209052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0bbff78c-22db-4ea3-8ba0-753c3046e692","Type":"ContainerStarted","Data":"9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e"}
Sep 09 20:28:37.209589 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:37.209134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0bbff78c-22db-4ea3-8ba0-753c3046e692","Type":"ContainerStarted","Data":"7380c2a43fbfef684c4480649fd76bcf36946026070656a3456b94903ac578e3"}
Sep 09 20:28:39.222397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:39.222358    2566 generic.go:356] "Generic (PLEG): container finished" podID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerID="9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e" exitCode=0
Sep 09 20:28:39.222828 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:39.222425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0bbff78c-22db-4ea3-8ba0-753c3046e692","Type":"ContainerDied","Data":"9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e"}
Sep 09 20:28:40.229644 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:40.229604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0bbff78c-22db-4ea3-8ba0-753c3046e692","Type":"ContainerStarted","Data":"f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2"}
Sep 09 20:28:40.251821 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:40.251754    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.957967363 podStartE2EDuration="5.251738543s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.29377118s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:40.249597389 +0000 UTC m=+1045.285068550" watchObservedRunningTime="2026-09-09 20:28:40.251738543 +0000 UTC m=+1045.287209703"
Sep 09 20:28:40.935547 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:40.935505    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:45.935680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:45.935639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:28:45.937244 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:45.937216    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:28:55.936322 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:28:55.936278    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:29:05.936103 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:05.936029    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:29:15.936064 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:15.936022    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:29:25.935506 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:25.935457    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:29:35.935730 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:35.935679    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:29:45.935644 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:45.935596    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:29:49.271104 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.271048    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"]
Sep 09 20:29:49.275102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.275060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.279634 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.279608    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 09 20:29:49.279765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.279607    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:29:49.279765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.279686    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 09 20:29:49.279765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.279691    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-xn56p\""
Sep 09 20:29:49.279900 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.279751    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 09 20:29:49.279900 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.279871    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 09 20:29:49.285574 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.285551    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"]
Sep 09 20:29:49.415794 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.415746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tmp-dir\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.415794 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.415805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kserve-provision-location\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.416006 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.415876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-home\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.416006 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.415927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-model-cache\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.416006 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.415956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-cabundle-cert\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.416131 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.416036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-dshm\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.416131 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.416074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tls-certs\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.416131 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.416122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7p9\" (UniqueName: \"kubernetes.io/projected/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kube-api-access-qw7p9\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.517690 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-model-cache\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.517898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-cabundle-cert\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.517898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-dshm\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.517898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tls-certs\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.517898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7p9\" (UniqueName: \"kubernetes.io/projected/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kube-api-access-qw7p9\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.517898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tmp-dir\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.517898 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kserve-provision-location\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.518299 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.517939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-home\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.518299 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.518107    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-model-cache\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.518401 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.518307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-home\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.518513 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.518489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tmp-dir\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.518626 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.518495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-cabundle-cert\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.518626 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.518580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kserve-provision-location\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.521455 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.521386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-dshm\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.521646 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.521630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tls-certs\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.529345 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.529314    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7p9\" (UniqueName: \"kubernetes.io/projected/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kube-api-access-qw7p9\") pod \"auth-enabled-test-kserve-67bb467bd8-5vqnz\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.588915 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.588885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:49.870245 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:49.870216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"]
Sep 09 20:29:49.871774 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:29:49.871740    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3252ec30_85c9_47ce_8ba7_4a0b025ecc5b.slice/crio-6b986231a505c4ec9542196279f6556fbe4323920e879522d7f75501e6c8456b WatchSource:0}: Error finding container 6b986231a505c4ec9542196279f6556fbe4323920e879522d7f75501e6c8456b: Status 404 returned error can't find the container with id 6b986231a505c4ec9542196279f6556fbe4323920e879522d7f75501e6c8456b
Sep 09 20:29:50.602878 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:50.602841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" event={"ID":"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b","Type":"ContainerStarted","Data":"b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa"}
Sep 09 20:29:50.603280 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:50.602902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" event={"ID":"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b","Type":"ContainerStarted","Data":"6b986231a505c4ec9542196279f6556fbe4323920e879522d7f75501e6c8456b"}
Sep 09 20:29:52.616729 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:52.616700    2566 generic.go:356] "Generic (PLEG): container finished" podID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerID="b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa" exitCode=0
Sep 09 20:29:52.617240 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:52.616785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" event={"ID":"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b","Type":"ContainerDied","Data":"b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa"}
Sep 09 20:29:53.624949 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:53.624907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" event={"ID":"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b","Type":"ContainerStarted","Data":"3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f"}
Sep 09 20:29:53.650274 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:53.650214    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podStartSLOduration=2.214265147 podStartE2EDuration="4.650197086s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.435931939s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:53.64683203 +0000 UTC m=+1118.682303192" watchObservedRunningTime="2026-09-09 20:29:53.650197086 +0000 UTC m=+1118.685668245"
Sep 09 20:29:54.589846 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:54.589754    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:55.936320 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:55.936274    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:29:59.589817 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:59.589774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:29:59.591643 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:29:59.591617    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:30:05.936102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:05.936039    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:30:09.589442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:09.589388    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:30:15.946134 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:15.946071    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:30:15.953806 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:15.953779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:30:19.589883 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:19.589841    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:30:21.815214 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:21.815173    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 20:30:29.589673 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:29.589630    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:30:36.840505 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:36.840394    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" containerID="cri-o://f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2" gracePeriod=45
Sep 09 20:30:36.946791 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:36.946746    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused"
Sep 09 20:30:37.603622 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.603596    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:30:37.756850 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.756762    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-tmp-dir\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.756850 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.756806    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-home\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.756850 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.756852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbff78c-22db-4ea3-8ba0-753c3046e692-tls-certs\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.757178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.756869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-dshm\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.757178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.756920    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54k9m\" (UniqueName: \"kubernetes.io/projected/0bbff78c-22db-4ea3-8ba0-753c3046e692-kube-api-access-54k9m\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.757178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.756943    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-model-cache\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.757178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.756986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-kserve-provision-location\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.757178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.757052    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbff78c-22db-4ea3-8ba0-753c3046e692-cabundle-cert\") pod \"0bbff78c-22db-4ea3-8ba0-753c3046e692\" (UID: \"0bbff78c-22db-4ea3-8ba0-753c3046e692\") "
Sep 09 20:30:37.757679 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.757646    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:37.757855 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.757674    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-home" (OuterVolumeSpecName: "home") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:37.757855 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.757733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbff78c-22db-4ea3-8ba0-753c3046e692-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:37.759918 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.759886    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-dshm" (OuterVolumeSpecName: "dshm") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:37.760014 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.759991    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbff78c-22db-4ea3-8ba0-753c3046e692-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:37.760060 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.760015    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbff78c-22db-4ea3-8ba0-753c3046e692-kube-api-access-54k9m" (OuterVolumeSpecName: "kube-api-access-54k9m") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "kube-api-access-54k9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:37.770454 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.770429    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:37.787155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.787115    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bbff78c-22db-4ea3-8ba0-753c3046e692" (UID: "0bbff78c-22db-4ea3-8ba0-753c3046e692"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:37.858723 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858691    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.858723 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858718    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.858723 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858727    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbff78c-22db-4ea3-8ba0-753c3046e692-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.858723 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858734    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.859195 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858744    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54k9m\" (UniqueName: \"kubernetes.io/projected/0bbff78c-22db-4ea3-8ba0-753c3046e692-kube-api-access-54k9m\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.859195 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858753    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.859195 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858761    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbff78c-22db-4ea3-8ba0-753c3046e692-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.859195 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.858769    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbff78c-22db-4ea3-8ba0-753c3046e692-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.874401 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.874371    2566 generic.go:356] "Generic (PLEG): container finished" podID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerID="f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2" exitCode=0
Sep 09 20:30:37.874559 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.874417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0bbff78c-22db-4ea3-8ba0-753c3046e692","Type":"ContainerDied","Data":"f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2"}
Sep 09 20:30:37.874559 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.874430    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 09 20:30:37.874559 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.874523    2566 scope.go:117] "RemoveContainer" containerID="f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2"
Sep 09 20:30:37.874719 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.874511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0bbff78c-22db-4ea3-8ba0-753c3046e692","Type":"ContainerDied","Data":"7380c2a43fbfef684c4480649fd76bcf36946026070656a3456b94903ac578e3"}
Sep 09 20:30:37.885173 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.885150    2566 scope.go:117] "RemoveContainer" containerID="9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e"
Sep 09 20:30:37.893369 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.893350    2566 scope.go:117] "RemoveContainer" containerID="f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2"
Sep 09 20:30:37.893690 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:30:37.893673    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2\": container with ID starting with f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2 not found: ID does not exist" containerID="f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2"
Sep 09 20:30:37.893754 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.893697    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2"} err="failed to get container status \"f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2\": rpc error: code = NotFound desc = could not find container \"f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2\": container with ID starting with f071cd6e30fdcd2bf0c7b7127a2bef1d5ba91aaf4c4151c14bae91b0500ddbf2 not found: ID does not exist"
Sep 09 20:30:37.893754 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.893713    2566 scope.go:117] "RemoveContainer" containerID="9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e"
Sep 09 20:30:37.893970 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:30:37.893951    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e\": container with ID starting with 9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e not found: ID does not exist" containerID="9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e"
Sep 09 20:30:37.894026 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.893980    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e"} err="failed to get container status \"9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e\": rpc error: code = NotFound desc = could not find container \"9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e\": container with ID starting with 9ec645c4b79de2dd05789813baa3182e353bc5bd108574505129ecccf117dc6e not found: ID does not exist"
Sep 09 20:30:37.897278 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.897253    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 20:30:37.902416 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:37.902393    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 09 20:30:39.492102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:39.492058    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" path="/var/lib/kubelet/pods/0bbff78c-22db-4ea3-8ba0-753c3046e692/volumes"
Sep 09 20:30:39.589931 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:39.589884    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:30:49.589750 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:49.589706    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:30:59.589619 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:30:59.589573    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:31:09.589998 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:31:09.589951    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:31:19.589639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:31:19.589594    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:31:29.589876 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:31:29.589821    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:31:39.590052 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:31:39.590005    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 09 20:31:49.599186 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:31:49.599149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:31:49.606715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:31:49.606693    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:31:54.870732 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:31:54.870694    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"]
Sep 09 20:32:09.896010 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:09.895899    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" containerID="cri-o://3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f" gracePeriod=45
Sep 09 20:32:10.644666 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.644639    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:32:10.658305 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658281    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-dshm\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.658426 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658338    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-home\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.658426 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658368    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw7p9\" (UniqueName: \"kubernetes.io/projected/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kube-api-access-qw7p9\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.658426 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kserve-provision-location\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.658584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658434    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tls-certs\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.658584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658461    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-cabundle-cert\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.658584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tmp-dir\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.658584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.658559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-model-cache\") pod \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\" (UID: \"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b\") "
Sep 09 20:32:10.659250 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.659151    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.659374 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.659253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-home" (OuterVolumeSpecName: "home") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.659792 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.659760    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:10.660899 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.660872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-dshm" (OuterVolumeSpecName: "dshm") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.661007 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.660902    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kube-api-access-qw7p9" (OuterVolumeSpecName: "kube-api-access-qw7p9") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "kube-api-access-qw7p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:10.661143 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.661123    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:10.675782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.675752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.680932 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.680906    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" (UID: "3252ec30-85c9-47ce-8ba7-4a0b025ecc5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.760010 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.759967    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.760010 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.760004    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.760010 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.760015    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.760287 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.760026    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.760287 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.760037    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw7p9\" (UniqueName: \"kubernetes.io/projected/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kube-api-access-qw7p9\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.760287 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.760048    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.760287 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.760059    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.760287 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:10.760069    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:32:11.333642 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.333607    2566 generic.go:356] "Generic (PLEG): container finished" podID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerID="3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f" exitCode=0
Sep 09 20:32:11.334038 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.333666    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"
Sep 09 20:32:11.334038 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.333681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" event={"ID":"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b","Type":"ContainerDied","Data":"3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f"}
Sep 09 20:32:11.334038 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.333752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" event={"ID":"3252ec30-85c9-47ce-8ba7-4a0b025ecc5b","Type":"ContainerDied","Data":"6b986231a505c4ec9542196279f6556fbe4323920e879522d7f75501e6c8456b"}
Sep 09 20:32:11.334038 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.333775    2566 scope.go:117] "RemoveContainer" containerID="3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f"
Sep 09 20:32:11.343019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.342999    2566 scope.go:117] "RemoveContainer" containerID="b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa"
Sep 09 20:32:11.350768 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.350751    2566 scope.go:117] "RemoveContainer" containerID="3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f"
Sep 09 20:32:11.351049 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:32:11.351031    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f\": container with ID starting with 3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f not found: ID does not exist" containerID="3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f"
Sep 09 20:32:11.351131 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.351058    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f"} err="failed to get container status \"3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f\": rpc error: code = NotFound desc = could not find container \"3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f\": container with ID starting with 3c64e55ec11b8345f9292b631e9719df0218e683e969399a83626822fcf4e82f not found: ID does not exist"
Sep 09 20:32:11.351131 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.351076    2566 scope.go:117] "RemoveContainer" containerID="b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa"
Sep 09 20:32:11.351345 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:32:11.351325    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa\": container with ID starting with b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa not found: ID does not exist" containerID="b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa"
Sep 09 20:32:11.351397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.351353    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa"} err="failed to get container status \"b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa\": rpc error: code = NotFound desc = could not find container \"b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa\": container with ID starting with b370ad2f4faa85a04ad0f1c0cab88ab6e9f8fa62d2f7c5d8a564d4c8fca032aa not found: ID does not exist"
Sep 09 20:32:11.356934 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.356913    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"]
Sep 09 20:32:11.361151 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.361127    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz"]
Sep 09 20:32:11.491491 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.491458    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" path="/var/lib/kubelet/pods/3252ec30-85c9-47ce-8ba7-4a0b025ecc5b/volumes"
Sep 09 20:32:11.600354 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:11.600259    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-67bb467bd8-5vqnz" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:32:54.969666 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.969634    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"]
Sep 09 20:32:54.970255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970150    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="storage-initializer"
Sep 09 20:32:54.970255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970170    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="storage-initializer"
Sep 09 20:32:54.970255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970184    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main"
Sep 09 20:32:54.970255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970192    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main"
Sep 09 20:32:54.970255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970211    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main"
Sep 09 20:32:54.970255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970219    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main"
Sep 09 20:32:54.970255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970255    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="storage-initializer"
Sep 09 20:32:54.970609 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970264    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="storage-initializer"
Sep 09 20:32:54.970609 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970363    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3252ec30-85c9-47ce-8ba7-4a0b025ecc5b" containerName="main"
Sep 09 20:32:54.970609 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.970376    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bbff78c-22db-4ea3-8ba0-753c3046e692" containerName="main"
Sep 09 20:32:54.973572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.973552    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:54.976336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.976304    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 09 20:32:54.976480 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.976384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 09 20:32:54.976480 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.976417    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 09 20:32:54.976480 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.976474    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-pwx87\""
Sep 09 20:32:54.976664 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.976545    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 09 20:32:54.976719 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.976678    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:32:54.982781 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:54.982759    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"]
Sep 09 20:32:55.137790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.137744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8rlw\" (UniqueName: \"kubernetes.io/projected/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kube-api-access-r8rlw\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.137790 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.137789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-dshm\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.138008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.137810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-model-cache\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.138008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.137876    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tls-certs\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.138008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.137951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tmp-dir\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.138008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.138001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3f577-dbfa-478c-bd39-8fb977c7469f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.138196 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.138018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-home\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.138196 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.138040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239220 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tls-certs\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tmp-dir\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3f577-dbfa-478c-bd39-8fb977c7469f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-home\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rlw\" (UniqueName: \"kubernetes.io/projected/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kube-api-access-r8rlw\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-dshm\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-model-cache\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tmp-dir\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239729    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-home\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.239864 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-model-cache\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.240038 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.239881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.240194 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.240178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3f577-dbfa-478c-bd39-8fb977c7469f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.241750 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.241723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-dshm\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.241851 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.241816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tls-certs\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.247901 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.247877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rlw\" (UniqueName: \"kubernetes.io/projected/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kube-api-access-r8rlw\") pod \"auth-invalid-token-test-kserve-6fb996ff97-rbwvf\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.285254 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.285220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:32:55.564994 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:55.564965    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"]
Sep 09 20:32:55.565991 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:32:55.565967    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf3f577_dbfa_478c_bd39_8fb977c7469f.slice/crio-f249b20d702322db4a24ebd77808d951e8ee6a83153290723cab46481e10ed97 WatchSource:0}: Error finding container f249b20d702322db4a24ebd77808d951e8ee6a83153290723cab46481e10ed97: Status 404 returned error can't find the container with id f249b20d702322db4a24ebd77808d951e8ee6a83153290723cab46481e10ed97
Sep 09 20:32:56.539365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:56.539329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" event={"ID":"fdf3f577-dbfa-478c-bd39-8fb977c7469f","Type":"ContainerStarted","Data":"b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b"}
Sep 09 20:32:56.539795 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:56.539385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" event={"ID":"fdf3f577-dbfa-478c-bd39-8fb977c7469f","Type":"ContainerStarted","Data":"f249b20d702322db4a24ebd77808d951e8ee6a83153290723cab46481e10ed97"}
Sep 09 20:32:58.551365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:58.551331    2566 generic.go:356] "Generic (PLEG): container finished" podID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerID="b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b" exitCode=0
Sep 09 20:32:58.551740 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:58.551411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" event={"ID":"fdf3f577-dbfa-478c-bd39-8fb977c7469f","Type":"ContainerDied","Data":"b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b"}
Sep 09 20:32:59.558484 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:59.558448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" event={"ID":"fdf3f577-dbfa-478c-bd39-8fb977c7469f","Type":"ContainerStarted","Data":"231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633"}
Sep 09 20:32:59.582041 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:32:59.581976    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podStartSLOduration=3.2724685510000002 podStartE2EDuration="5.581957476s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.309488925s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:59.580722687 +0000 UTC m=+1304.616193856" watchObservedRunningTime="2026-09-09 20:32:59.581957476 +0000 UTC m=+1304.617428637"
Sep 09 20:33:00.285536 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:00.285493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:33:05.286062 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:05.286016    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:33:05.287747 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:05.287712    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:33:15.286219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:15.286169    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:33:25.285917 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:25.285861    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:33:35.285999 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:35.285944    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:33:45.285935 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:45.285844    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:33:55.286283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:33:55.286238    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:34:05.285807 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:05.285766    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:34:15.286215 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:15.286169    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:34:25.286140 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:25.286060    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:34:35.286101 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:35.286049    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:34:45.286127 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:45.286055    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:34:55.296206 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:55.296172    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:34:55.304627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:55.304594    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:34:56.977881 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:34:56.977848    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"]
Sep 09 20:35:12.181467 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:12.181354    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" containerID="cri-o://231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633" gracePeriod=45
Sep 09 20:35:12.297247 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:12.297201    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 09 20:35:12.934722 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:12.934692    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:35:13.079021 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.078925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-home\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079021 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.078978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8rlw\" (UniqueName: \"kubernetes.io/projected/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kube-api-access-r8rlw\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079021 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.078995    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-dshm\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079344 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079056    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-model-cache\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079344 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079074    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tmp-dir\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079344 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3f577-dbfa-478c-bd39-8fb977c7469f-cabundle-cert\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079344 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079148    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tls-certs\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079344 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079182    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kserve-provision-location\") pod \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\" (UID: \"fdf3f577-dbfa-478c-bd39-8fb977c7469f\") "
Sep 09 20:35:13.079603 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079581    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:13.079675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079658    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf3f577-dbfa-478c-bd39-8fb977c7469f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:13.079774 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.079747    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-home" (OuterVolumeSpecName: "home") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:13.082274 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.082245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-dshm" (OuterVolumeSpecName: "dshm") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:13.082274 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.082246    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kube-api-access-r8rlw" (OuterVolumeSpecName: "kube-api-access-r8rlw") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "kube-api-access-r8rlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:13.082433 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.082411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:13.092397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.092369    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:13.104637 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.104607    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdf3f577-dbfa-478c-bd39-8fb977c7469f" (UID: "fdf3f577-dbfa-478c-bd39-8fb977c7469f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:13.180691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180660    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.180691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180689    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3f577-dbfa-478c-bd39-8fb977c7469f-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.180691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180698    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf3f577-dbfa-478c-bd39-8fb977c7469f-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.180907 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180708    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.180907 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180717    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.180907 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180727    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8rlw\" (UniqueName: \"kubernetes.io/projected/fdf3f577-dbfa-478c-bd39-8fb977c7469f-kube-api-access-r8rlw\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.180907 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180735    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.180907 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.180742    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf3f577-dbfa-478c-bd39-8fb977c7469f-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:35:13.231491 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.231460    2566 generic.go:356] "Generic (PLEG): container finished" podID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerID="231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633" exitCode=0
Sep 09 20:35:13.231939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.231504    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"
Sep 09 20:35:13.231939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.231539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" event={"ID":"fdf3f577-dbfa-478c-bd39-8fb977c7469f","Type":"ContainerDied","Data":"231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633"}
Sep 09 20:35:13.231939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.231619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf" event={"ID":"fdf3f577-dbfa-478c-bd39-8fb977c7469f","Type":"ContainerDied","Data":"f249b20d702322db4a24ebd77808d951e8ee6a83153290723cab46481e10ed97"}
Sep 09 20:35:13.231939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.231654    2566 scope.go:117] "RemoveContainer" containerID="231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633"
Sep 09 20:35:13.240911 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.240892    2566 scope.go:117] "RemoveContainer" containerID="b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b"
Sep 09 20:35:13.249238 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.249213    2566 scope.go:117] "RemoveContainer" containerID="231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633"
Sep 09 20:35:13.249678 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:35:13.249659    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633\": container with ID starting with 231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633 not found: ID does not exist" containerID="231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633"
Sep 09 20:35:13.249751 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.249686    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633"} err="failed to get container status \"231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633\": rpc error: code = NotFound desc = could not find container \"231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633\": container with ID starting with 231f54a146839a1a1a13f2afaf430138fdc9fd02c2af36b7c7f86da3b4943633 not found: ID does not exist"
Sep 09 20:35:13.249751 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.249704    2566 scope.go:117] "RemoveContainer" containerID="b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b"
Sep 09 20:35:13.249970 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:35:13.249952    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b\": container with ID starting with b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b not found: ID does not exist" containerID="b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b"
Sep 09 20:35:13.250006 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.249976    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b"} err="failed to get container status \"b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b\": rpc error: code = NotFound desc = could not find container \"b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b\": container with ID starting with b117eb7b2522e83894fa9fe193bc24fbb9c0fa8161a73a440f07a3c45d4f013b not found: ID does not exist"
Sep 09 20:35:13.256793 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.256766    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"]
Sep 09 20:35:13.261025 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.260998    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6fb996ff97-rbwvf"]
Sep 09 20:35:13.490989 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:13.490957    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" path="/var/lib/kubelet/pods/fdf3f577-dbfa-478c-bd39-8fb977c7469f/volumes"
Sep 09 20:35:52.454389 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.454355    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"]
Sep 09 20:35:52.454953 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.454817    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="storage-initializer"
Sep 09 20:35:52.454953 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.454832    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="storage-initializer"
Sep 09 20:35:52.454953 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.454842    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main"
Sep 09 20:35:52.454953 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.454848    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main"
Sep 09 20:35:52.454953 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.454904    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf3f577-dbfa-478c-bd39-8fb977c7469f" containerName="main"
Sep 09 20:35:52.458408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.458376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.461423 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.461401    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 09 20:35:52.461553 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.461449    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:35:52.461553 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.461483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-tq5zs\""
Sep 09 20:35:52.461683 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.461666    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 09 20:35:52.461843 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.461803    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 09 20:35:52.461928 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.461869    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 09 20:35:52.471724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.471697    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"]
Sep 09 20:35:52.520812 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.520778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24986887-023d-477e-8c74-68d9d1b833b3-tls-certs\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.520812 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.520823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24986887-023d-477e-8c74-68d9d1b833b3-cabundle-cert\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.521108 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.520849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-kserve-provision-location\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.521108 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.520898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-dshm\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.521108 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.520927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkg6\" (UniqueName: \"kubernetes.io/projected/24986887-023d-477e-8c74-68d9d1b833b3-kube-api-access-ghkg6\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.521108 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.520991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-home\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.521108 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.521029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-model-cache\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.521108 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.521067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-tmp-dir\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622099 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-dshm\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622273 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkg6\" (UniqueName: \"kubernetes.io/projected/24986887-023d-477e-8c74-68d9d1b833b3-kube-api-access-ghkg6\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622273 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-home\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622273 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-model-cache\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622273 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-tmp-dir\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622480 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24986887-023d-477e-8c74-68d9d1b833b3-tls-certs\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622480 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24986887-023d-477e-8c74-68d9d1b833b3-cabundle-cert\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622637 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-home\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622637 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-model-cache\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622838 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-kserve-provision-location\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622838 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-tmp-dir\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.622943 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-kserve-provision-location\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.623002 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.622968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24986887-023d-477e-8c74-68d9d1b833b3-cabundle-cert\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.624573 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.624553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-dshm\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.624921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.624904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24986887-023d-477e-8c74-68d9d1b833b3-tls-certs\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.631918 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.631879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkg6\" (UniqueName: \"kubernetes.io/projected/24986887-023d-477e-8c74-68d9d1b833b3-kube-api-access-ghkg6\") pod \"auth-disabled-test-kserve-7cc6d5b986-qmnkb\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:52.769312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:52.769224    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:35:53.049239 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:53.049163    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"]
Sep 09 20:35:53.052818 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:35:53.052795    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24986887_023d_477e_8c74_68d9d1b833b3.slice/crio-916d0200ad8c5002a7a6621934868a8590b033ca52ba52c9a07ea2150034fbed WatchSource:0}: Error finding container 916d0200ad8c5002a7a6621934868a8590b033ca52ba52c9a07ea2150034fbed: Status 404 returned error can't find the container with id 916d0200ad8c5002a7a6621934868a8590b033ca52ba52c9a07ea2150034fbed
Sep 09 20:35:53.421859 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:53.421809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" event={"ID":"24986887-023d-477e-8c74-68d9d1b833b3","Type":"ContainerStarted","Data":"44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7"}
Sep 09 20:35:53.422045 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:53.421882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" event={"ID":"24986887-023d-477e-8c74-68d9d1b833b3","Type":"ContainerStarted","Data":"916d0200ad8c5002a7a6621934868a8590b033ca52ba52c9a07ea2150034fbed"}
Sep 09 20:35:55.433143 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:55.433112    2566 generic.go:356] "Generic (PLEG): container finished" podID="24986887-023d-477e-8c74-68d9d1b833b3" containerID="44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7" exitCode=0
Sep 09 20:35:55.433497 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:55.433183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" event={"ID":"24986887-023d-477e-8c74-68d9d1b833b3","Type":"ContainerDied","Data":"44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7"}
Sep 09 20:35:56.440718 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:56.440684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" event={"ID":"24986887-023d-477e-8c74-68d9d1b833b3","Type":"ContainerStarted","Data":"c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178"}
Sep 09 20:35:56.466345 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:56.466282    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podStartSLOduration=2.197663239 podStartE2EDuration="4.466261693s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.268598454s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:56.465419744 +0000 UTC m=+1481.500890903" watchObservedRunningTime="2026-09-09 20:35:56.466261693 +0000 UTC m=+1481.501732859"
Sep 09 20:35:56.770010 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:35:56.769930    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:36:02.769907 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:02.769863    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:36:02.771368 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:02.771338    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:36:12.770350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:12.770307    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:36:22.560443 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.560405    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"]
Sep 09 20:36:22.564273 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.564246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:22.567116 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.567051    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-zzqnc\""
Sep 09 20:36:22.568056 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.568028    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 09 20:36:22.568213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.568053    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 09 20:36:22.568213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.568031    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 09 20:36:22.568213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.568030    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-kl5vr\""
Sep 09 20:36:22.572683 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.572652    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"]
Sep 09 20:36:22.590944 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.590906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a2e86b-7dea-4939-b226-818be6fda183-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:22.591148 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.590960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gq5k\" (UniqueName: \"kubernetes.io/projected/b7a2e86b-7dea-4939-b226-818be6fda183-kube-api-access-5gq5k\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:22.691882 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.691845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a2e86b-7dea-4939-b226-818be6fda183-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:22.692074 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.691891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gq5k\" (UniqueName: \"kubernetes.io/projected/b7a2e86b-7dea-4939-b226-818be6fda183-kube-api-access-5gq5k\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:22.694752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.694726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a2e86b-7dea-4939-b226-818be6fda183-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:22.701560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.701530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gq5k\" (UniqueName: \"kubernetes.io/projected/b7a2e86b-7dea-4939-b226-818be6fda183-kube-api-access-5gq5k\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:22.769655 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.769609    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:36:22.878705 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:22.878669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:23.159579 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:23.159548    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"]
Sep 09 20:36:23.161862 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:36:23.161834    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a2e86b_7dea_4939_b226_818be6fda183.slice/crio-a30492eb045d75e1e83f0ed400079ec3783bf7b569843e65137982dc0d0d5cad WatchSource:0}: Error finding container a30492eb045d75e1e83f0ed400079ec3783bf7b569843e65137982dc0d0d5cad: Status 404 returned error can't find the container with id a30492eb045d75e1e83f0ed400079ec3783bf7b569843e65137982dc0d0d5cad
Sep 09 20:36:23.164756 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:23.164734    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:36:23.574519 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:23.574438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" event={"ID":"b7a2e86b-7dea-4939-b226-818be6fda183","Type":"ContainerStarted","Data":"a30492eb045d75e1e83f0ed400079ec3783bf7b569843e65137982dc0d0d5cad"}
Sep 09 20:36:26.593647 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:26.593609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" event={"ID":"b7a2e86b-7dea-4939-b226-818be6fda183","Type":"ContainerStarted","Data":"5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d"}
Sep 09 20:36:26.594148 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:26.593738    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:36:26.611437 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:26.611379    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" podStartSLOduration=1.673741765 podStartE2EDuration="4.611356271s" totalImagesPullingTime="2.937614506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:26.610663618 +0000 UTC m=+1511.646134808" watchObservedRunningTime="2026-09-09 20:36:26.611356271 +0000 UTC m=+1511.646827431"
Sep 09 20:36:32.770449 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:32.770402    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:36:42.770668 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:42.770572    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:36:52.770419 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:52.770364    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:36:56.538416 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:56.538378    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"]
Sep 09 20:36:56.540407 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:36:56.540371    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:37:02.769761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:02.769710    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:37:06.541074 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:06.541029    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:37:11.559972 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:11.559898    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main" containerID="cri-o://5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d" gracePeriod=45
Sep 09 20:37:12.769899 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:12.769853    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:37:16.540801 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:16.540757    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:37:22.770336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:22.770293    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:37:26.540742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:26.540693    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:37:32.769992 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:32.769936    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:37:36.540494 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:36.540443    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:37:41.813942 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.813913    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:37:41.928734 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.928694    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gq5k\" (UniqueName: \"kubernetes.io/projected/b7a2e86b-7dea-4939-b226-818be6fda183-kube-api-access-5gq5k\") pod \"b7a2e86b-7dea-4939-b226-818be6fda183\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") "
Sep 09 20:37:41.928935 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.928753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a2e86b-7dea-4939-b226-818be6fda183-tls-certs\") pod \"b7a2e86b-7dea-4939-b226-818be6fda183\" (UID: \"b7a2e86b-7dea-4939-b226-818be6fda183\") "
Sep 09 20:37:41.931112 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.931062    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a2e86b-7dea-4939-b226-818be6fda183-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7a2e86b-7dea-4939-b226-818be6fda183" (UID: "b7a2e86b-7dea-4939-b226-818be6fda183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:41.931242 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.931162    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a2e86b-7dea-4939-b226-818be6fda183-kube-api-access-5gq5k" (OuterVolumeSpecName: "kube-api-access-5gq5k") pod "b7a2e86b-7dea-4939-b226-818be6fda183" (UID: "b7a2e86b-7dea-4939-b226-818be6fda183"). InnerVolumeSpecName "kube-api-access-5gq5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:41.972515 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.972476    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7a2e86b-7dea-4939-b226-818be6fda183" containerID="5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d" exitCode=0
Sep 09 20:37:41.972680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.972573    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"
Sep 09 20:37:41.972680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.972614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" event={"ID":"b7a2e86b-7dea-4939-b226-818be6fda183","Type":"ContainerDied","Data":"5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d"}
Sep 09 20:37:41.972680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.972672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9" event={"ID":"b7a2e86b-7dea-4939-b226-818be6fda183","Type":"ContainerDied","Data":"a30492eb045d75e1e83f0ed400079ec3783bf7b569843e65137982dc0d0d5cad"}
Sep 09 20:37:41.972784 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.972701    2566 scope.go:117] "RemoveContainer" containerID="5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d"
Sep 09 20:37:41.982219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.982200    2566 scope.go:117] "RemoveContainer" containerID="5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d"
Sep 09 20:37:41.982541 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:37:41.982521    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d\": container with ID starting with 5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d not found: ID does not exist" containerID="5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d"
Sep 09 20:37:41.982592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.982549    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d"} err="failed to get container status \"5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d\": rpc error: code = NotFound desc = could not find container \"5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d\": container with ID starting with 5ba2210dae7c2ad4035514b68e3bfac2553e5bcbc7fcf1bac92766b37b89f80d not found: ID does not exist"
Sep 09 20:37:41.994797 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.994767    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"]
Sep 09 20:37:41.998924 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:41.998897    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qhzw9"]
Sep 09 20:37:42.029608 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:42.029581    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gq5k\" (UniqueName: \"kubernetes.io/projected/b7a2e86b-7dea-4939-b226-818be6fda183-kube-api-access-5gq5k\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:37:42.029608 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:42.029607    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a2e86b-7dea-4939-b226-818be6fda183-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:37:42.770621 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:42.770576    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:37:43.491239 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:43.491202    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" path="/var/lib/kubelet/pods/b7a2e86b-7dea-4939-b226-818be6fda183/volumes"
Sep 09 20:37:52.780331 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:52.780297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:37:52.788976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:37:52.788945    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:38:19.621445 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:19.621405    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"]
Sep 09 20:38:34.645655 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:34.645588    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" containerID="cri-o://c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178" gracePeriod=45
Sep 09 20:38:34.781996 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:34.781953    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Sep 09 20:38:35.396126 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.396100    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:38:35.506135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506039    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24986887-023d-477e-8c74-68d9d1b833b3-cabundle-cert\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506069    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-home\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24986887-023d-477e-8c74-68d9d1b833b3-tls-certs\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-tmp-dir\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506169    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-model-cache\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-dshm\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghkg6\" (UniqueName: \"kubernetes.io/projected/24986887-023d-477e-8c74-68d9d1b833b3-kube-api-access-ghkg6\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506287    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-kserve-provision-location\") pod \"24986887-023d-477e-8c74-68d9d1b833b3\" (UID: \"24986887-023d-477e-8c74-68d9d1b833b3\") "
Sep 09 20:38:35.506639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24986887-023d-477e-8c74-68d9d1b833b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:35.506639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:35.506639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506529    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24986887-023d-477e-8c74-68d9d1b833b3-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:35.506802 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.506724    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-home" (OuterVolumeSpecName: "home") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:35.508717 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.508692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-dshm" (OuterVolumeSpecName: "dshm") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:35.508717 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.508703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24986887-023d-477e-8c74-68d9d1b833b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:35.508900 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.508871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24986887-023d-477e-8c74-68d9d1b833b3-kube-api-access-ghkg6" (OuterVolumeSpecName: "kube-api-access-ghkg6") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "kube-api-access-ghkg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:35.524796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.524770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:35.536910 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.536877    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24986887-023d-477e-8c74-68d9d1b833b3" (UID: "24986887-023d-477e-8c74-68d9d1b833b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:35.606984 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.606959    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:35.606984 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.606985    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24986887-023d-477e-8c74-68d9d1b833b3-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:35.607203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.606998    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:35.607203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.607015    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:35.607203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.607024    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:35.607203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.607038    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghkg6\" (UniqueName: \"kubernetes.io/projected/24986887-023d-477e-8c74-68d9d1b833b3-kube-api-access-ghkg6\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:35.607203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:35.607050    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24986887-023d-477e-8c74-68d9d1b833b3-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:38:36.232256 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.232219    2566 generic.go:356] "Generic (PLEG): container finished" podID="24986887-023d-477e-8c74-68d9d1b833b3" containerID="c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178" exitCode=0
Sep 09 20:38:36.232716 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.232277    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"
Sep 09 20:38:36.232716 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.232310    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" event={"ID":"24986887-023d-477e-8c74-68d9d1b833b3","Type":"ContainerDied","Data":"c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178"}
Sep 09 20:38:36.232716 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.232380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb" event={"ID":"24986887-023d-477e-8c74-68d9d1b833b3","Type":"ContainerDied","Data":"916d0200ad8c5002a7a6621934868a8590b033ca52ba52c9a07ea2150034fbed"}
Sep 09 20:38:36.232716 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.232413    2566 scope.go:117] "RemoveContainer" containerID="c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178"
Sep 09 20:38:36.242365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.242349    2566 scope.go:117] "RemoveContainer" containerID="44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7"
Sep 09 20:38:36.249956 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.249939    2566 scope.go:117] "RemoveContainer" containerID="c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178"
Sep 09 20:38:36.250247 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:38:36.250227    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178\": container with ID starting with c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178 not found: ID does not exist" containerID="c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178"
Sep 09 20:38:36.250300 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.250257    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178"} err="failed to get container status \"c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178\": rpc error: code = NotFound desc = could not find container \"c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178\": container with ID starting with c1e04d4ef3cd340bc2564287d97c621487a3ba5cba5aeed334fe91f0decd6178 not found: ID does not exist"
Sep 09 20:38:36.250300 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.250272    2566 scope.go:117] "RemoveContainer" containerID="44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7"
Sep 09 20:38:36.250528 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:38:36.250510    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7\": container with ID starting with 44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7 not found: ID does not exist" containerID="44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7"
Sep 09 20:38:36.250568 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.250534    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7"} err="failed to get container status \"44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7\": rpc error: code = NotFound desc = could not find container \"44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7\": container with ID starting with 44a4e7da96b8d77f7e05fa377333bfeb3f9b7b7ad4bf98e8df8581d7c9e608f7 not found: ID does not exist"
Sep 09 20:38:36.254833 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.254813    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"]
Sep 09 20:38:36.260328 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:36.260307    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7cc6d5b986-qmnkb"]
Sep 09 20:38:37.491991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:38:37.491960    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24986887-023d-477e-8c74-68d9d1b833b3" path="/var/lib/kubelet/pods/24986887-023d-477e-8c74-68d9d1b833b3/volumes"
Sep 09 20:39:51.497299 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497222    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"]
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497639    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main"
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497656    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main"
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497677    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="storage-initializer"
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497683    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="storage-initializer"
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497691    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main"
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497696    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main"
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497753    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="24986887-023d-477e-8c74-68d9d1b833b3" containerName="main"
Sep 09 20:39:51.497779 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.497761    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a2e86b-7dea-4939-b226-818be6fda183" containerName="main"
Sep 09 20:39:51.500712 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.500690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.504768 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.504741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 09 20:39:51.504872 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.504742    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 09 20:39:51.504872 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.504741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 09 20:39:51.504968 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.504745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-jqxss\""
Sep 09 20:39:51.509921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.509899    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"]
Sep 09 20:39:51.638018 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.637984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.638203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.638021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.638203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.638062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.638203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.638108    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a6233b-c9dc-4a16-8f2d-39160a695343-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.638203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.638145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.638203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.638200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87fc\" (UniqueName: \"kubernetes.io/projected/95a6233b-c9dc-4a16-8f2d-39160a695343-kube-api-access-j87fc\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.738861 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.738829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.738861 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.738865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a6233b-c9dc-4a16-8f2d-39160a695343-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.739070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.738887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.739070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.738949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87fc\" (UniqueName: \"kubernetes.io/projected/95a6233b-c9dc-4a16-8f2d-39160a695343-kube-api-access-j87fc\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.739070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.739000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.739070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.739028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.739415 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.739387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.739510 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.739416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.739510 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.739451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.741252 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.741236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.741639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.741618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a6233b-c9dc-4a16-8f2d-39160a695343-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.748621 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.748561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87fc\" (UniqueName: \"kubernetes.io/projected/95a6233b-c9dc-4a16-8f2d-39160a695343-kube-api-access-j87fc\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:51.811507 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:51.811476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:39:52.083576 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:52.083539    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"]
Sep 09 20:39:52.085791 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:39:52.085762    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a6233b_c9dc_4a16_8f2d_39160a695343.slice/crio-1c29f2cd0ca477eabce4d647389fd442c1255763ea0871f8294e5b294a36c08f WatchSource:0}: Error finding container 1c29f2cd0ca477eabce4d647389fd442c1255763ea0871f8294e5b294a36c08f: Status 404 returned error can't find the container with id 1c29f2cd0ca477eabce4d647389fd442c1255763ea0871f8294e5b294a36c08f
Sep 09 20:39:52.583380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:52.583343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" event={"ID":"95a6233b-c9dc-4a16-8f2d-39160a695343","Type":"ContainerStarted","Data":"1c29f2cd0ca477eabce4d647389fd442c1255763ea0871f8294e5b294a36c08f"}
Sep 09 20:39:54.594471 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:54.594432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" event={"ID":"95a6233b-c9dc-4a16-8f2d-39160a695343","Type":"ContainerStarted","Data":"7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21"}
Sep 09 20:39:54.612728 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:54.612679    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" podStartSLOduration=2.090858787 podStartE2EDuration="3.612663643s" totalImagesPullingTime="1.521804856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:54.610770247 +0000 UTC m=+1719.646241433" watchObservedRunningTime="2026-09-09 20:39:54.612663643 +0000 UTC m=+1719.648134802"
Sep 09 20:39:54.812054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:39:54.811991    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:40:01.812027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:01.811989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:40:01.824830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:01.824801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:40:02.640766 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:02.640736    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:40:31.611418 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:31.611379    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"]
Sep 09 20:40:31.620961 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:40:31.620927    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:40:31.620961 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:31.620961 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:31.620961 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21" cmd=["/bin/sleep","15"]
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:31.620991    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" message="" err=<
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:40:31.621006    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" podUID="95a6233b-c9dc-4a16-8f2d-39160a695343" containerName="main" containerID="cri-o://7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21"
Sep 09 20:40:31.621272 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:31.621051    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" podUID="95a6233b-c9dc-4a16-8f2d-39160a695343" containerName="main" containerID="cri-o://7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21" gracePeriod=60
Sep 09 20:40:31.630851 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:31.630816    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" podUID="95a6233b-c9dc-4a16-8f2d-39160a695343" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 09 20:40:31.976397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:31.976371    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:40:32.095428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095393    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-dshm\") pod \"95a6233b-c9dc-4a16-8f2d-39160a695343\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") "
Sep 09 20:40:32.095670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-tmp-dir\") pod \"95a6233b-c9dc-4a16-8f2d-39160a695343\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") "
Sep 09 20:40:32.095670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j87fc\" (UniqueName: \"kubernetes.io/projected/95a6233b-c9dc-4a16-8f2d-39160a695343-kube-api-access-j87fc\") pod \"95a6233b-c9dc-4a16-8f2d-39160a695343\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") "
Sep 09 20:40:32.095670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-model-cache\") pod \"95a6233b-c9dc-4a16-8f2d-39160a695343\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") "
Sep 09 20:40:32.095670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a6233b-c9dc-4a16-8f2d-39160a695343-tls-certs\") pod \"95a6233b-c9dc-4a16-8f2d-39160a695343\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") "
Sep 09 20:40:32.095670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095546    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-home\") pod \"95a6233b-c9dc-4a16-8f2d-39160a695343\" (UID: \"95a6233b-c9dc-4a16-8f2d-39160a695343\") "
Sep 09 20:40:32.095964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-home" (OuterVolumeSpecName: "home") pod "95a6233b-c9dc-4a16-8f2d-39160a695343" (UID: "95a6233b-c9dc-4a16-8f2d-39160a695343"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.095964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.095936    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-model-cache" (OuterVolumeSpecName: "model-cache") pod "95a6233b-c9dc-4a16-8f2d-39160a695343" (UID: "95a6233b-c9dc-4a16-8f2d-39160a695343"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.096070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.096010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95a6233b-c9dc-4a16-8f2d-39160a695343" (UID: "95a6233b-c9dc-4a16-8f2d-39160a695343"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.098251 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.098224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-dshm" (OuterVolumeSpecName: "dshm") pod "95a6233b-c9dc-4a16-8f2d-39160a695343" (UID: "95a6233b-c9dc-4a16-8f2d-39160a695343"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:32.098251 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.098237    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a6233b-c9dc-4a16-8f2d-39160a695343-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95a6233b-c9dc-4a16-8f2d-39160a695343" (UID: "95a6233b-c9dc-4a16-8f2d-39160a695343"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:32.098405 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.098247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a6233b-c9dc-4a16-8f2d-39160a695343-kube-api-access-j87fc" (OuterVolumeSpecName: "kube-api-access-j87fc") pod "95a6233b-c9dc-4a16-8f2d-39160a695343" (UID: "95a6233b-c9dc-4a16-8f2d-39160a695343"). InnerVolumeSpecName "kube-api-access-j87fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:32.196533 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.196458    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.196533 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.196483    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.196533 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.196499    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j87fc\" (UniqueName: \"kubernetes.io/projected/95a6233b-c9dc-4a16-8f2d-39160a695343-kube-api-access-j87fc\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.196533 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.196511    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.196533 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.196519    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95a6233b-c9dc-4a16-8f2d-39160a695343-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.196533 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.196526    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95a6233b-c9dc-4a16-8f2d-39160a695343-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.771709 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.771670    2566 generic.go:356] "Generic (PLEG): container finished" podID="95a6233b-c9dc-4a16-8f2d-39160a695343" containerID="7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21" exitCode=0
Sep 09 20:40:32.772147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.771715    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"
Sep 09 20:40:32.772147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.771757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" event={"ID":"95a6233b-c9dc-4a16-8f2d-39160a695343","Type":"ContainerDied","Data":"7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21"}
Sep 09 20:40:32.772147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.771811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz" event={"ID":"95a6233b-c9dc-4a16-8f2d-39160a695343","Type":"ContainerDied","Data":"1c29f2cd0ca477eabce4d647389fd442c1255763ea0871f8294e5b294a36c08f"}
Sep 09 20:40:32.772147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.771834    2566 scope.go:117] "RemoveContainer" containerID="7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21"
Sep 09 20:40:32.781606 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.781586    2566 scope.go:117] "RemoveContainer" containerID="7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21"
Sep 09 20:40:32.781909 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:40:32.781890    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21\": container with ID starting with 7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21 not found: ID does not exist" containerID="7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21"
Sep 09 20:40:32.781962 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.781919    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21"} err="failed to get container status \"7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21\": rpc error: code = NotFound desc = could not find container \"7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21\": container with ID starting with 7681dc890fe63ec3961b5d2a8f1a7220bbaa8e47d28e911b621e72547e961a21 not found: ID does not exist"
Sep 09 20:40:32.795415 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.795387    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"]
Sep 09 20:40:32.799568 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:32.799539    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-zc4sz"]
Sep 09 20:40:33.490784 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:40:33.490751    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a6233b-c9dc-4a16-8f2d-39160a695343" path="/var/lib/kubelet/pods/95a6233b-c9dc-4a16-8f2d-39160a695343/volumes"
Sep 09 20:41:50.225766 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.225728    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"]
Sep 09 20:41:50.226739 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.226706    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a6233b-c9dc-4a16-8f2d-39160a695343" containerName="main"
Sep 09 20:41:50.226739 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.226734    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a6233b-c9dc-4a16-8f2d-39160a695343" containerName="main"
Sep 09 20:41:50.227199 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.226820    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a6233b-c9dc-4a16-8f2d-39160a695343" containerName="main"
Sep 09 20:41:50.229731 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.229715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.232243 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.232219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 09 20:41:50.232383 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.232257    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-spczr\""
Sep 09 20:41:50.233219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.233198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 09 20:41:50.233313 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.233247    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 09 20:41:50.238496 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.238472    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"]
Sep 09 20:41:50.273984 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.273954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.274172 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.274003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.274172 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.274074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.274172 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.274136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.274283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.274200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptpk\" (UniqueName: \"kubernetes.io/projected/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-kube-api-access-sptpk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.274283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.274221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375388 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375596 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375596 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375596 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375596 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sptpk\" (UniqueName: \"kubernetes.io/projected/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-kube-api-access-sptpk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375596 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375863 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375863 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.375964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.375909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.377842 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.377816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.378076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.378059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.384902 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.384878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptpk\" (UniqueName: \"kubernetes.io/projected/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-kube-api-access-sptpk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.542151 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.542030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:41:50.821291 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:50.821261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"]
Sep 09 20:41:50.822837 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:41:50.822810    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eb34c3_4774_4923_b7c8_c6adb5c10fd0.slice/crio-e2d55b145615e3f5e0d7a47fb022876faa0a032fa2359eec143247cdf23fbb1c WatchSource:0}: Error finding container e2d55b145615e3f5e0d7a47fb022876faa0a032fa2359eec143247cdf23fbb1c: Status 404 returned error can't find the container with id e2d55b145615e3f5e0d7a47fb022876faa0a032fa2359eec143247cdf23fbb1c
Sep 09 20:41:51.126932 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:51.126896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" event={"ID":"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0","Type":"ContainerStarted","Data":"c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98"}
Sep 09 20:41:51.127122 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:51.126952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" event={"ID":"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0","Type":"ContainerStarted","Data":"e2d55b145615e3f5e0d7a47fb022876faa0a032fa2359eec143247cdf23fbb1c"}
Sep 09 20:41:51.145510 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:51.145456    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" podStartSLOduration=1.145438781 podStartE2EDuration="1.145438781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:51.144613842 +0000 UTC m=+1836.180085002" watchObservedRunningTime="2026-09-09 20:41:51.145438781 +0000 UTC m=+1836.180909941"
Sep 09 20:41:51.543006 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:41:51.542929    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:42:00.543066 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:00.543026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:42:00.555939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:00.555910    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:42:01.183578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:01.183549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:42:25.848906 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.848860    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"]
Sep 09 20:42:25.852263 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.852244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.855671 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.855642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 09 20:42:25.855810 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.855672    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 09 20:42:25.855810 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.855674    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 09 20:42:25.855810 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.855648    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-fwvl4\""
Sep 09 20:42:25.863250 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.863226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"]
Sep 09 20:42:25.890545 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.890511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8nmk\" (UniqueName: \"kubernetes.io/projected/2f9e96db-c13e-445f-b5ae-400d7ea02343-kube-api-access-w8nmk\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.890752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.890569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.890752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.890630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9e96db-c13e-445f-b5ae-400d7ea02343-tls-certs\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.890752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.890666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-dshm\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.890752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.890721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-model-cache\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.890752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.890751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-home\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9e96db-c13e-445f-b5ae-400d7ea02343-tls-certs\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991569 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-dshm\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991569 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-model-cache\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991569 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-home\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991569 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8nmk\" (UniqueName: \"kubernetes.io/projected/2f9e96db-c13e-445f-b5ae-400d7ea02343-kube-api-access-w8nmk\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991775 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991872 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-model-cache\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.991932 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.991857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-home\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.993868 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.993843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-dshm\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:25.994060 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:25.994044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9e96db-c13e-445f-b5ae-400d7ea02343-tls-certs\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:26.000878 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:26.000856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8nmk\" (UniqueName: \"kubernetes.io/projected/2f9e96db-c13e-445f-b5ae-400d7ea02343-kube-api-access-w8nmk\") pod \"test-div-v1-kserve-fb566d667-r5bkl\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:26.164318 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:26.164276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:26.466859 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:26.466832    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"]
Sep 09 20:42:26.469187 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:42:26.469156    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9e96db_c13e_445f_b5ae_400d7ea02343.slice/crio-77caa2b928e8821cbbccc6449bf01aa355930fed51014a8a31afb7e95da3d952 WatchSource:0}: Error finding container 77caa2b928e8821cbbccc6449bf01aa355930fed51014a8a31afb7e95da3d952: Status 404 returned error can't find the container with id 77caa2b928e8821cbbccc6449bf01aa355930fed51014a8a31afb7e95da3d952
Sep 09 20:42:26.471155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:26.471130    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:42:27.292893 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:27.292855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl" event={"ID":"2f9e96db-c13e-445f-b5ae-400d7ea02343","Type":"ContainerStarted","Data":"77caa2b928e8821cbbccc6449bf01aa355930fed51014a8a31afb7e95da3d952"}
Sep 09 20:42:28.299126 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:28.299074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl" event={"ID":"2f9e96db-c13e-445f-b5ae-400d7ea02343","Type":"ContainerStarted","Data":"e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676"}
Sep 09 20:42:28.320171 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:28.320101    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl" podStartSLOduration=2.174862483 podStartE2EDuration="3.320061057s" totalImagesPullingTime="1.145198574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:28.317460463 +0000 UTC m=+1873.352931624" watchObservedRunningTime="2026-09-09 20:42:28.320061057 +0000 UTC m=+1873.355532219"
Sep 09 20:42:29.165153 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:29.165114    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:36.165224 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:36.165185    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:36.177506 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:36.177481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:42:36.347015 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:42:36.346987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:43:30.413112 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.413062    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"]
Sep 09 20:43:30.422555 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:43:30.422514    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:43:30.422555 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:43:30.422555 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:43:30.422555 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98" cmd=["/bin/sleep","15"]
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.422579    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" message="" err=<
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:43:30.422598    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" podUID="b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" containerName="main" containerID="cri-o://c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98"
Sep 09 20:43:30.422801 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.422652    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" podUID="b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" containerName="main" containerID="cri-o://c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98" gracePeriod=60
Sep 09 20:43:30.770169 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.770139    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:43:30.865661 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.865621    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-dshm\") pod \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") "
Sep 09 20:43:30.865860 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.865678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sptpk\" (UniqueName: \"kubernetes.io/projected/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-kube-api-access-sptpk\") pod \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") "
Sep 09 20:43:30.865860 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.865701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tls-certs\") pod \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") "
Sep 09 20:43:30.865860 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.865747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-model-cache\") pod \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") "
Sep 09 20:43:30.865860 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.865792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-home\") pod \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") "
Sep 09 20:43:30.866135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.865874    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tmp-dir\") pod \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\" (UID: \"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0\") "
Sep 09 20:43:30.866135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.866064    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" (UID: "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:30.866247 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.866135    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-home" (OuterVolumeSpecName: "home") pod "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" (UID: "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:30.866247 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.866220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" (UID: "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:30.866334 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.866229    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:30.866334 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.866277    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:30.868168 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.868147    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-dshm" (OuterVolumeSpecName: "dshm") pod "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" (UID: "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:30.868258 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.868242    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-kube-api-access-sptpk" (OuterVolumeSpecName: "kube-api-access-sptpk") pod "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" (UID: "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0"). InnerVolumeSpecName "kube-api-access-sptpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:30.868310 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.868250    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" (UID: "b5eb34c3-4774-4923-b7c8-c6adb5c10fd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:30.967230 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.967145    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:30.967230 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.967170    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:30.967230 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.967179    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sptpk\" (UniqueName: \"kubernetes.io/projected/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-kube-api-access-sptpk\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:30.967230 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:30.967188    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:31.591336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.591299    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" containerID="c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98" exitCode=0
Sep 09 20:43:31.591803 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.591352    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"
Sep 09 20:43:31.591803 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.591389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" event={"ID":"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0","Type":"ContainerDied","Data":"c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98"}
Sep 09 20:43:31.591803 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.591447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm" event={"ID":"b5eb34c3-4774-4923-b7c8-c6adb5c10fd0","Type":"ContainerDied","Data":"e2d55b145615e3f5e0d7a47fb022876faa0a032fa2359eec143247cdf23fbb1c"}
Sep 09 20:43:31.591803 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.591483    2566 scope.go:117] "RemoveContainer" containerID="c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98"
Sep 09 20:43:31.600658 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.600631    2566 scope.go:117] "RemoveContainer" containerID="c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98"
Sep 09 20:43:31.600918 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:43:31.600899    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98\": container with ID starting with c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98 not found: ID does not exist" containerID="c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98"
Sep 09 20:43:31.600972 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.600931    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98"} err="failed to get container status \"c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98\": rpc error: code = NotFound desc = could not find container \"c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98\": container with ID starting with c571d54096c55785c573df4a49a631719574175e8e80c60fa8fd19c813c63f98 not found: ID does not exist"
Sep 09 20:43:31.608593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.608572    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"]
Sep 09 20:43:31.614679 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:31.614660    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64447v2sm"]
Sep 09 20:43:33.492365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:33.492336    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" path="/var/lib/kubelet/pods/b5eb34c3-4774-4923-b7c8-c6adb5c10fd0/volumes"
Sep 09 20:43:36.703895 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:36.699507    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"]
Sep 09 20:43:51.732738 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:51.732679    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl" podUID="2f9e96db-c13e-445f-b5ae-400d7ea02343" containerName="main" containerID="cri-o://e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676" gracePeriod=45
Sep 09 20:43:51.982974 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:51.982918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:43:52.148558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9e96db-c13e-445f-b5ae-400d7ea02343-tls-certs\") pod \"2f9e96db-c13e-445f-b5ae-400d7ea02343\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") "
Sep 09 20:43:52.148762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148600    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-tmp-dir\") pod \"2f9e96db-c13e-445f-b5ae-400d7ea02343\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") "
Sep 09 20:43:52.148762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8nmk\" (UniqueName: \"kubernetes.io/projected/2f9e96db-c13e-445f-b5ae-400d7ea02343-kube-api-access-w8nmk\") pod \"2f9e96db-c13e-445f-b5ae-400d7ea02343\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") "
Sep 09 20:43:52.148762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148722    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-model-cache\") pod \"2f9e96db-c13e-445f-b5ae-400d7ea02343\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") "
Sep 09 20:43:52.148762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148756    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-home\") pod \"2f9e96db-c13e-445f-b5ae-400d7ea02343\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") "
Sep 09 20:43:52.148979 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-dshm\") pod \"2f9e96db-c13e-445f-b5ae-400d7ea02343\" (UID: \"2f9e96db-c13e-445f-b5ae-400d7ea02343\") "
Sep 09 20:43:52.148979 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148876    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f9e96db-c13e-445f-b5ae-400d7ea02343" (UID: "2f9e96db-c13e-445f-b5ae-400d7ea02343"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.148979 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.148940    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f9e96db-c13e-445f-b5ae-400d7ea02343" (UID: "2f9e96db-c13e-445f-b5ae-400d7ea02343"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.149185 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.149076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-home" (OuterVolumeSpecName: "home") pod "2f9e96db-c13e-445f-b5ae-400d7ea02343" (UID: "2f9e96db-c13e-445f-b5ae-400d7ea02343"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.149242 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.149211    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.149344 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.149244    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.149344 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.149260    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.151159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.151127    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9e96db-c13e-445f-b5ae-400d7ea02343-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f9e96db-c13e-445f-b5ae-400d7ea02343" (UID: "2f9e96db-c13e-445f-b5ae-400d7ea02343"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:52.151277 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.151152    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-dshm" (OuterVolumeSpecName: "dshm") pod "2f9e96db-c13e-445f-b5ae-400d7ea02343" (UID: "2f9e96db-c13e-445f-b5ae-400d7ea02343"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:52.151277 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.151233    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9e96db-c13e-445f-b5ae-400d7ea02343-kube-api-access-w8nmk" (OuterVolumeSpecName: "kube-api-access-w8nmk") pod "2f9e96db-c13e-445f-b5ae-400d7ea02343" (UID: "2f9e96db-c13e-445f-b5ae-400d7ea02343"). InnerVolumeSpecName "kube-api-access-w8nmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:52.249934 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.249872    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9e96db-c13e-445f-b5ae-400d7ea02343-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.249934 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.249896    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9e96db-c13e-445f-b5ae-400d7ea02343-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.249934 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.249908    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8nmk\" (UniqueName: \"kubernetes.io/projected/2f9e96db-c13e-445f-b5ae-400d7ea02343-kube-api-access-w8nmk\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:43:52.691152 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.691119    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f9e96db-c13e-445f-b5ae-400d7ea02343" containerID="e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676" exitCode=0
Sep 09 20:43:52.691327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.691180    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"
Sep 09 20:43:52.691327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.691214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl" event={"ID":"2f9e96db-c13e-445f-b5ae-400d7ea02343","Type":"ContainerDied","Data":"e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676"}
Sep 09 20:43:52.691327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.691269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl" event={"ID":"2f9e96db-c13e-445f-b5ae-400d7ea02343","Type":"ContainerDied","Data":"77caa2b928e8821cbbccc6449bf01aa355930fed51014a8a31afb7e95da3d952"}
Sep 09 20:43:52.691327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.691290    2566 scope.go:117] "RemoveContainer" containerID="e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676"
Sep 09 20:43:52.700028 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.700013    2566 scope.go:117] "RemoveContainer" containerID="e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676"
Sep 09 20:43:52.700320 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:43:52.700296    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676\": container with ID starting with e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676 not found: ID does not exist" containerID="e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676"
Sep 09 20:43:52.700422 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.700333    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676"} err="failed to get container status \"e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676\": rpc error: code = NotFound desc = could not find container \"e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676\": container with ID starting with e488baa5c25781039126926cdef6164ad6268535c278134f30ef4688a9ed5676 not found: ID does not exist"
Sep 09 20:43:52.715866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.715839    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"]
Sep 09 20:43:52.722261 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:52.722237    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-r5bkl"]
Sep 09 20:43:53.446909 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.446876    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"]
Sep 09 20:43:53.447324 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.447225    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9e96db-c13e-445f-b5ae-400d7ea02343" containerName="main"
Sep 09 20:43:53.447324 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.447239    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9e96db-c13e-445f-b5ae-400d7ea02343" containerName="main"
Sep 09 20:43:53.447324 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.447262    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" containerName="main"
Sep 09 20:43:53.447324 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.447268    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" containerName="main"
Sep 09 20:43:53.447324 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.447327    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9e96db-c13e-445f-b5ae-400d7ea02343" containerName="main"
Sep 09 20:43:53.447522 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.447337    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5eb34c3-4774-4923-b7c8-c6adb5c10fd0" containerName="main"
Sep 09 20:43:53.451769 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.451749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.455307 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.455276    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-7csxv\""
Sep 09 20:43:53.455442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.455325    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 09 20:43:53.455442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.455347    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 09 20:43:53.455442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.455349    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 09 20:43:53.462225 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.462203    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"]
Sep 09 20:43:53.493683 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.493653    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9e96db-c13e-445f-b5ae-400d7ea02343" path="/var/lib/kubelet/pods/2f9e96db-c13e-445f-b5ae-400d7ea02343/volumes"
Sep 09 20:43:53.560446 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.560412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.560584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.560451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-model-cache\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.560584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.560518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-home\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.560584 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.560542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-dshm\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.560691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.560641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpfn\" (UniqueName: \"kubernetes.io/projected/7b64e47e-5474-4dea-b631-e09f8e0eeee2-kube-api-access-mjpfn\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.560691 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.560685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tls-certs\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.661701 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.661656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-model-cache\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.661887 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.661739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-home\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.661887 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.661770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-dshm\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.661887 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.661826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpfn\" (UniqueName: \"kubernetes.io/projected/7b64e47e-5474-4dea-b631-e09f8e0eeee2-kube-api-access-mjpfn\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.661887 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.661852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tls-certs\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.662120 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.661901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.662190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.662144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-model-cache\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.662190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.662166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-home\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.662304 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.662279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.664310 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.664289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-dshm\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.664453 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.664436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tls-certs\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.670751 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.670726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpfn\" (UniqueName: \"kubernetes.io/projected/7b64e47e-5474-4dea-b631-e09f8e0eeee2-kube-api-access-mjpfn\") pod \"stop-v1-kserve-5657b88c95-p5gfv\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:53.764178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:53.764102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:43:54.040069 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:54.040042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"]
Sep 09 20:43:54.043207 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:43:54.043177    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b64e47e_5474_4dea_b631_e09f8e0eeee2.slice/crio-35761803e6e747a28a1bb38a07667abc98235a8f28732041992d3b96740a2067 WatchSource:0}: Error finding container 35761803e6e747a28a1bb38a07667abc98235a8f28732041992d3b96740a2067: Status 404 returned error can't find the container with id 35761803e6e747a28a1bb38a07667abc98235a8f28732041992d3b96740a2067
Sep 09 20:43:54.704249 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:54.704216    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv" event={"ID":"7b64e47e-5474-4dea-b631-e09f8e0eeee2","Type":"ContainerStarted","Data":"bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9"}
Sep 09 20:43:54.704703 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:54.704277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv" event={"ID":"7b64e47e-5474-4dea-b631-e09f8e0eeee2","Type":"ContainerStarted","Data":"35761803e6e747a28a1bb38a07667abc98235a8f28732041992d3b96740a2067"}
Sep 09 20:43:54.725343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:54.725285    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv" podStartSLOduration=1.7252700060000001 podStartE2EDuration="1.725270006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:54.723291146 +0000 UTC m=+1959.758762317" watchObservedRunningTime="2026-09-09 20:43:54.725270006 +0000 UTC m=+1959.760741194"
Sep 09 20:43:54.764744 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:43:54.764716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:44:03.765042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:03.765007    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:44:03.777905 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:03.777881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:44:04.760867 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:04.760832    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:44:20.752308 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.752274    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"]
Sep 09 20:44:20.755654 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.755635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.758202 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.758174    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 09 20:44:20.765783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.765753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"]
Sep 09 20:44:20.776781 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.776758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.776915 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.776792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-home\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.776915 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.776830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-model-cache\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.776915 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.776880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5n7\" (UniqueName: \"kubernetes.io/projected/295eb84c-e6fa-46da-9e75-fab5d29d2853-kube-api-access-bt5n7\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.777111 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.776973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295eb84c-e6fa-46da-9e75-fab5d29d2853-tls-certs\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.777111 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.777022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-dshm\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.877839 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.877795    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.877839 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.877838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-home\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.878136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.877865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-model-cache\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.878136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.877895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5n7\" (UniqueName: \"kubernetes.io/projected/295eb84c-e6fa-46da-9e75-fab5d29d2853-kube-api-access-bt5n7\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.878136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.877951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295eb84c-e6fa-46da-9e75-fab5d29d2853-tls-certs\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.878136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.878007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-dshm\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.878355 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.878281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-home\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.878355 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.878316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-model-cache\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.878473 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.878352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.880482 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.880460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-dshm\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.880634 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.880613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295eb84c-e6fa-46da-9e75-fab5d29d2853-tls-certs\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:20.887349 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:20.887322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5n7\" (UniqueName: \"kubernetes.io/projected/295eb84c-e6fa-46da-9e75-fab5d29d2853-kube-api-access-bt5n7\") pod \"stop-v2-kserve-7f68747c45-lrv9b\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:21.068489 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:21.068402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:21.345904 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:21.345822    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"]
Sep 09 20:44:21.349778 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:44:21.349737    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295eb84c_e6fa_46da_9e75_fab5d29d2853.slice/crio-4e9cca4e2da8016cdff79f34c57e7528940d4c3663fd658352a240f3a8d567da WatchSource:0}: Error finding container 4e9cca4e2da8016cdff79f34c57e7528940d4c3663fd658352a240f3a8d567da: Status 404 returned error can't find the container with id 4e9cca4e2da8016cdff79f34c57e7528940d4c3663fd658352a240f3a8d567da
Sep 09 20:44:21.834616 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:21.834582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b" event={"ID":"295eb84c-e6fa-46da-9e75-fab5d29d2853","Type":"ContainerStarted","Data":"feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9"}
Sep 09 20:44:21.835008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:21.834633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b" event={"ID":"295eb84c-e6fa-46da-9e75-fab5d29d2853","Type":"ContainerStarted","Data":"4e9cca4e2da8016cdff79f34c57e7528940d4c3663fd658352a240f3a8d567da"}
Sep 09 20:44:21.853146 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:21.853078    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b" podStartSLOduration=1.8530622719999998 podStartE2EDuration="1.853062272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:21.850848113 +0000 UTC m=+1986.886319272" watchObservedRunningTime="2026-09-09 20:44:21.853062272 +0000 UTC m=+1986.888533432"
Sep 09 20:44:22.069151 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:22.069118    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:31.068875 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:31.068825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:31.081667 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:31.081632    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:31.890930 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:31.890901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:44:38.561625 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.561591    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"]
Sep 09 20:44:38.565350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.565332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.568294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.568270    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-pngj7\""
Sep 09 20:44:38.568427 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.568311    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:44:38.569150 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.569135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 09 20:44:38.569229 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.569157    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 09 20:44:38.569229 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.569198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 09 20:44:38.569229 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.569207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 09 20:44:38.574689 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.574668    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"]
Sep 09 20:44:38.629994 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.629968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.630153 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.629996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.630153 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.630013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m84g\" (UniqueName: \"kubernetes.io/projected/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kube-api-access-9m84g\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.630275 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.630177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-dshm\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.630275 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.630266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-model-cache\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.630388 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.630290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ed047-63e9-4855-ad2b-685e06e90c7a-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.630388 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.630339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-home\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.630388 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.630375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.730977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-dshm\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-model-cache\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ed047-63e9-4855-ad2b-685e06e90c7a-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-home\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m84g\" (UniqueName: \"kubernetes.io/projected/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kube-api-access-9m84g\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731380 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-model-cache\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731549 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731705 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-home\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.731761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.731747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ed047-63e9-4855-ad2b-685e06e90c7a-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.733395 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.733377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-dshm\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.733653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.733635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.741857 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.741837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m84g\" (UniqueName: \"kubernetes.io/projected/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kube-api-access-9m84g\") pod \"tokenizer-clean-path-test-kserve-9b65d76bd-6jp22\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:38.877296 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:38.877255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:39.160098 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:39.160051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"]
Sep 09 20:44:39.161600 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:44:39.161566    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2ed047_63e9_4855_ad2b_685e06e90c7a.slice/crio-640a8f69d648a2c91f6afdfd45903071c073b7422682cc2726907fcd3c51549d WatchSource:0}: Error finding container 640a8f69d648a2c91f6afdfd45903071c073b7422682cc2726907fcd3c51549d: Status 404 returned error can't find the container with id 640a8f69d648a2c91f6afdfd45903071c073b7422682cc2726907fcd3c51549d
Sep 09 20:44:39.918899 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:39.918864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" event={"ID":"dc2ed047-63e9-4855-ad2b-685e06e90c7a","Type":"ContainerStarted","Data":"9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc"}
Sep 09 20:44:39.919372 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:39.918916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" event={"ID":"dc2ed047-63e9-4855-ad2b-685e06e90c7a","Type":"ContainerStarted","Data":"640a8f69d648a2c91f6afdfd45903071c073b7422682cc2726907fcd3c51549d"}
Sep 09 20:44:41.930237 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:41.930199    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerID="9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc" exitCode=0
Sep 09 20:44:41.930628 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:41.930271    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" event={"ID":"dc2ed047-63e9-4855-ad2b-685e06e90c7a","Type":"ContainerDied","Data":"9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc"}
Sep 09 20:44:42.937701 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:42.937665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" event={"ID":"dc2ed047-63e9-4855-ad2b-685e06e90c7a","Type":"ContainerStarted","Data":"a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44"}
Sep 09 20:44:42.961868 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:42.961801    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" podStartSLOduration=2.713133879 podStartE2EDuration="4.961777739s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.24864386s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:42.958652915 +0000 UTC m=+2007.994124111" watchObservedRunningTime="2026-09-09 20:44:42.961777739 +0000 UTC m=+2007.997248899"
Sep 09 20:44:43.877977 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:43.877944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:48.877543 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:48.877497    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:48.896434 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:48.896401    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:44:48.977117 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:44:48.977070    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:45:13.427434 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:13.427396    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"]
Sep 09 20:45:28.447958 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.447909    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b" podUID="295eb84c-e6fa-46da-9e75-fab5d29d2853" containerName="main" containerID="cri-o://feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9" gracePeriod=45
Sep 09 20:45:28.690688 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.690664    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:45:28.769742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.769669    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-model-cache\") pod \"295eb84c-e6fa-46da-9e75-fab5d29d2853\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") "
Sep 09 20:45:28.769921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.769742    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-home\") pod \"295eb84c-e6fa-46da-9e75-fab5d29d2853\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") "
Sep 09 20:45:28.769921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.769781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-tmp-dir\") pod \"295eb84c-e6fa-46da-9e75-fab5d29d2853\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") "
Sep 09 20:45:28.769921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.769826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt5n7\" (UniqueName: \"kubernetes.io/projected/295eb84c-e6fa-46da-9e75-fab5d29d2853-kube-api-access-bt5n7\") pod \"295eb84c-e6fa-46da-9e75-fab5d29d2853\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") "
Sep 09 20:45:28.769921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.769850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295eb84c-e6fa-46da-9e75-fab5d29d2853-tls-certs\") pod \"295eb84c-e6fa-46da-9e75-fab5d29d2853\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") "
Sep 09 20:45:28.769921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.769900    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-dshm\") pod \"295eb84c-e6fa-46da-9e75-fab5d29d2853\" (UID: \"295eb84c-e6fa-46da-9e75-fab5d29d2853\") "
Sep 09 20:45:28.770241 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.769954    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-model-cache" (OuterVolumeSpecName: "model-cache") pod "295eb84c-e6fa-46da-9e75-fab5d29d2853" (UID: "295eb84c-e6fa-46da-9e75-fab5d29d2853"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:28.770241 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.770032    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-home" (OuterVolumeSpecName: "home") pod "295eb84c-e6fa-46da-9e75-fab5d29d2853" (UID: "295eb84c-e6fa-46da-9e75-fab5d29d2853"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:28.770241 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.770044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "295eb84c-e6fa-46da-9e75-fab5d29d2853" (UID: "295eb84c-e6fa-46da-9e75-fab5d29d2853"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:28.770241 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.770217    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:28.770241 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.770234    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:28.770425 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.770248    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:28.772349 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.772317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295eb84c-e6fa-46da-9e75-fab5d29d2853-kube-api-access-bt5n7" (OuterVolumeSpecName: "kube-api-access-bt5n7") pod "295eb84c-e6fa-46da-9e75-fab5d29d2853" (UID: "295eb84c-e6fa-46da-9e75-fab5d29d2853"). InnerVolumeSpecName "kube-api-access-bt5n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:28.772349 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.772330    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-dshm" (OuterVolumeSpecName: "dshm") pod "295eb84c-e6fa-46da-9e75-fab5d29d2853" (UID: "295eb84c-e6fa-46da-9e75-fab5d29d2853"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:28.772491 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.772352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295eb84c-e6fa-46da-9e75-fab5d29d2853-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "295eb84c-e6fa-46da-9e75-fab5d29d2853" (UID: "295eb84c-e6fa-46da-9e75-fab5d29d2853"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:28.870795 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.870753    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295eb84c-e6fa-46da-9e75-fab5d29d2853-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:28.870795 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.870787    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5n7\" (UniqueName: \"kubernetes.io/projected/295eb84c-e6fa-46da-9e75-fab5d29d2853-kube-api-access-bt5n7\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:28.870795 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:28.870800    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295eb84c-e6fa-46da-9e75-fab5d29d2853-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:29.151573 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.151539    2566 generic.go:356] "Generic (PLEG): container finished" podID="295eb84c-e6fa-46da-9e75-fab5d29d2853" containerID="feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9" exitCode=0
Sep 09 20:45:29.151749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.151585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"
Sep 09 20:45:29.151749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.151626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b" event={"ID":"295eb84c-e6fa-46da-9e75-fab5d29d2853","Type":"ContainerDied","Data":"feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9"}
Sep 09 20:45:29.151749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.151674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b" event={"ID":"295eb84c-e6fa-46da-9e75-fab5d29d2853","Type":"ContainerDied","Data":"4e9cca4e2da8016cdff79f34c57e7528940d4c3663fd658352a240f3a8d567da"}
Sep 09 20:45:29.151749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.151700    2566 scope.go:117] "RemoveContainer" containerID="feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9"
Sep 09 20:45:29.161374 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.161362    2566 scope.go:117] "RemoveContainer" containerID="feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9"
Sep 09 20:45:29.161646 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:45:29.161630    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9\": container with ID starting with feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9 not found: ID does not exist" containerID="feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9"
Sep 09 20:45:29.161696 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.161666    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9"} err="failed to get container status \"feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9\": rpc error: code = NotFound desc = could not find container \"feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9\": container with ID starting with feef44bc5780b37cdbf79115c5fe064ea4d593a290f4e096617a83a830521ad9 not found: ID does not exist"
Sep 09 20:45:29.175238 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.175215    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"]
Sep 09 20:45:29.180613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.180592    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-lrv9b"]
Sep 09 20:45:29.492540 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:29.492450    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295eb84c-e6fa-46da-9e75-fab5d29d2853" path="/var/lib/kubelet/pods/295eb84c-e6fa-46da-9e75-fab5d29d2853/volumes"
Sep 09 20:45:41.707136 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:41.707040    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"]
Sep 09 20:45:50.546657 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.546624    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"]
Sep 09 20:45:50.547201 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.547029    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295eb84c-e6fa-46da-9e75-fab5d29d2853" containerName="main"
Sep 09 20:45:50.547201 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.547043    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="295eb84c-e6fa-46da-9e75-fab5d29d2853" containerName="main"
Sep 09 20:45:50.547201 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.547127    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="295eb84c-e6fa-46da-9e75-fab5d29d2853" containerName="main"
Sep 09 20:45:50.550266 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.550245    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.553891 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.553866    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 09 20:45:50.554021 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.553918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-l7mwr\""
Sep 09 20:45:50.554021 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.553874    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 09 20:45:50.554193 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.553866    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 09 20:45:50.560419 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.560395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"]
Sep 09 20:45:50.678238 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.678195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.678238 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.678241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-home\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.678478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.678277    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xrp\" (UniqueName: \"kubernetes.io/projected/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-kube-api-access-m8xrp\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.678478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.678312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.678478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.678393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-dshm\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.678478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.678447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-model-cache\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779163 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-dshm\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-model-cache\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-home\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xrp\" (UniqueName: \"kubernetes.io/projected/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-kube-api-access-m8xrp\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779663 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-home\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779663 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-model-cache\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.779776 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.779697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.781665 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.781643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-dshm\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.781950 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.781933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.792074 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.792046    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xrp\" (UniqueName: \"kubernetes.io/projected/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-kube-api-access-m8xrp\") pod \"decom-v1-kserve-6cb797fb75-sx2ll\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:50.862031 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:50.861998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:51.143599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:51.143570    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"]
Sep 09 20:45:51.146571 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:45:51.146540    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7185dc82_3fbe_4658_bbc7_4fb1b2b7bddc.slice/crio-a4658805f3396e96c8ed5c1b373ed8fe1f6c60675ae9bb9ea01dbd4cd5f60bde WatchSource:0}: Error finding container a4658805f3396e96c8ed5c1b373ed8fe1f6c60675ae9bb9ea01dbd4cd5f60bde: Status 404 returned error can't find the container with id a4658805f3396e96c8ed5c1b373ed8fe1f6c60675ae9bb9ea01dbd4cd5f60bde
Sep 09 20:45:51.255145 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:51.255106    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll" event={"ID":"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc","Type":"ContainerStarted","Data":"843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c"}
Sep 09 20:45:51.255304 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:51.255170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll" event={"ID":"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc","Type":"ContainerStarted","Data":"a4658805f3396e96c8ed5c1b373ed8fe1f6c60675ae9bb9ea01dbd4cd5f60bde"}
Sep 09 20:45:51.863022 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:51.862992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:45:56.727205 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:56.727163    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv" podUID="7b64e47e-5474-4dea-b631-e09f8e0eeee2" containerName="main" containerID="cri-o://bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9" gracePeriod=45
Sep 09 20:45:56.750199 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:56.750165    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv" podUID="7b64e47e-5474-4dea-b631-e09f8e0eeee2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 09 20:45:56.980229 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:56.980161    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:45:57.000439 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.000391    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll" podStartSLOduration=7.000373257 podStartE2EDuration="7.000373257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:51.291160405 +0000 UTC m=+2076.326631565" watchObservedRunningTime="2026-09-09 20:45:57.000373257 +0000 UTC m=+2082.035844417"
Sep 09 20:45:57.138617 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.138570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-home\") pod \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") "
Sep 09 20:45:57.138822 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.138652    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjpfn\" (UniqueName: \"kubernetes.io/projected/7b64e47e-5474-4dea-b631-e09f8e0eeee2-kube-api-access-mjpfn\") pod \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") "
Sep 09 20:45:57.138822 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.138714    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tmp-dir\") pod \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") "
Sep 09 20:45:57.138822 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.138736    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-model-cache\") pod \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") "
Sep 09 20:45:57.138822 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.138760    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-dshm\") pod \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") "
Sep 09 20:45:57.138822 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.138805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tls-certs\") pod \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\" (UID: \"7b64e47e-5474-4dea-b631-e09f8e0eeee2\") "
Sep 09 20:45:57.139062 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.138907    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-home" (OuterVolumeSpecName: "home") pod "7b64e47e-5474-4dea-b631-e09f8e0eeee2" (UID: "7b64e47e-5474-4dea-b631-e09f8e0eeee2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:57.139062 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.139049    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b64e47e-5474-4dea-b631-e09f8e0eeee2" (UID: "7b64e47e-5474-4dea-b631-e09f8e0eeee2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:57.139198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.139058    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b64e47e-5474-4dea-b631-e09f8e0eeee2" (UID: "7b64e47e-5474-4dea-b631-e09f8e0eeee2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:57.139198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.139146    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:57.139198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.139164    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:57.139198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.139173    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:57.141001 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.140971    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-dshm" (OuterVolumeSpecName: "dshm") pod "7b64e47e-5474-4dea-b631-e09f8e0eeee2" (UID: "7b64e47e-5474-4dea-b631-e09f8e0eeee2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:57.141132 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.141059    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b64e47e-5474-4dea-b631-e09f8e0eeee2-kube-api-access-mjpfn" (OuterVolumeSpecName: "kube-api-access-mjpfn") pod "7b64e47e-5474-4dea-b631-e09f8e0eeee2" (UID: "7b64e47e-5474-4dea-b631-e09f8e0eeee2"). InnerVolumeSpecName "kube-api-access-mjpfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:57.141132 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.141069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b64e47e-5474-4dea-b631-e09f8e0eeee2" (UID: "7b64e47e-5474-4dea-b631-e09f8e0eeee2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:57.240343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.240245    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b64e47e-5474-4dea-b631-e09f8e0eeee2-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:57.240343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.240287    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b64e47e-5474-4dea-b631-e09f8e0eeee2-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:57.240343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.240298    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjpfn\" (UniqueName: \"kubernetes.io/projected/7b64e47e-5474-4dea-b631-e09f8e0eeee2-kube-api-access-mjpfn\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:45:57.284004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.283963    2566 generic.go:356] "Generic (PLEG): container finished" podID="7b64e47e-5474-4dea-b631-e09f8e0eeee2" containerID="bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9" exitCode=0
Sep 09 20:45:57.284191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.284013    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"
Sep 09 20:45:57.284191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.284048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv" event={"ID":"7b64e47e-5474-4dea-b631-e09f8e0eeee2","Type":"ContainerDied","Data":"bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9"}
Sep 09 20:45:57.284191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.284119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv" event={"ID":"7b64e47e-5474-4dea-b631-e09f8e0eeee2","Type":"ContainerDied","Data":"35761803e6e747a28a1bb38a07667abc98235a8f28732041992d3b96740a2067"}
Sep 09 20:45:57.284191 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.284141    2566 scope.go:117] "RemoveContainer" containerID="bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9"
Sep 09 20:45:57.293739 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.293720    2566 scope.go:117] "RemoveContainer" containerID="bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9"
Sep 09 20:45:57.294018 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:45:57.293995    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9\": container with ID starting with bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9 not found: ID does not exist" containerID="bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9"
Sep 09 20:45:57.294101 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.294029    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9"} err="failed to get container status \"bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9\": rpc error: code = NotFound desc = could not find container \"bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9\": container with ID starting with bedd375b2bc3888cb7701fbfa826428460d25f95f31df49f5e77e4abdf0152d9 not found: ID does not exist"
Sep 09 20:45:57.307288 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.307257    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"]
Sep 09 20:45:57.312139 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.312117    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-p5gfv"]
Sep 09 20:45:57.491197 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:45:57.491118    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b64e47e-5474-4dea-b631-e09f8e0eeee2" path="/var/lib/kubelet/pods/7b64e47e-5474-4dea-b631-e09f8e0eeee2/volumes"
Sep 09 20:46:00.862832 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:00.862794    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:46:00.875355 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:00.875324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:46:01.316021 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:01.315933    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:46:08.048147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.048113    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"]
Sep 09 20:46:08.048532 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.048495    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b64e47e-5474-4dea-b631-e09f8e0eeee2" containerName="main"
Sep 09 20:46:08.048532 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.048506    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b64e47e-5474-4dea-b631-e09f8e0eeee2" containerName="main"
Sep 09 20:46:08.048617 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.048568    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b64e47e-5474-4dea-b631-e09f8e0eeee2" containerName="main"
Sep 09 20:46:08.051766 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.051747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.054546 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.054518    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 09 20:46:08.062267 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.062243    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"]
Sep 09 20:46:08.137003 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.136970    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da45b73-8860-4d58-9e5e-be2774f7aab0-tls-certs\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.137003 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.137003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-tmp-dir\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.137213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.137030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-home\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.137213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.137101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-dshm\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.137213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.137166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9pg2\" (UniqueName: \"kubernetes.io/projected/5da45b73-8860-4d58-9e5e-be2774f7aab0-kube-api-access-x9pg2\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.137213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.137200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-model-cache\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238232 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9pg2\" (UniqueName: \"kubernetes.io/projected/5da45b73-8860-4d58-9e5e-be2774f7aab0-kube-api-access-x9pg2\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238232 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-model-cache\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da45b73-8860-4d58-9e5e-be2774f7aab0-tls-certs\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-tmp-dir\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-home\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-dshm\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238730 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-model-cache\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238809 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-tmp-dir\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.238809 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.238779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-home\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.240755 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.240734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-dshm\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.240982 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.240965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da45b73-8860-4d58-9e5e-be2774f7aab0-tls-certs\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.250050 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.250028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9pg2\" (UniqueName: \"kubernetes.io/projected/5da45b73-8860-4d58-9e5e-be2774f7aab0-kube-api-access-x9pg2\") pod \"decom-v2-kserve-77f867d787-hhgd9\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.363673 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.363642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:08.637056 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:08.637033    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"]
Sep 09 20:46:08.638368 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:46:08.638335    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da45b73_8860_4d58_9e5e_be2774f7aab0.slice/crio-aa5e725804b479e725c6bcfd8db4b6b93e4fac68437db2863a85d28cb8bc164c WatchSource:0}: Error finding container aa5e725804b479e725c6bcfd8db4b6b93e4fac68437db2863a85d28cb8bc164c: Status 404 returned error can't find the container with id aa5e725804b479e725c6bcfd8db4b6b93e4fac68437db2863a85d28cb8bc164c
Sep 09 20:46:09.342653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:09.342618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9" event={"ID":"5da45b73-8860-4d58-9e5e-be2774f7aab0","Type":"ContainerStarted","Data":"c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0"}
Sep 09 20:46:09.343035 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:09.342671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9" event={"ID":"5da45b73-8860-4d58-9e5e-be2774f7aab0","Type":"ContainerStarted","Data":"aa5e725804b479e725c6bcfd8db4b6b93e4fac68437db2863a85d28cb8bc164c"}
Sep 09 20:46:09.363069 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:09.363021    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9" podStartSLOduration=1.363006668 podStartE2EDuration="1.363006668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:09.360923443 +0000 UTC m=+2094.396394602" watchObservedRunningTime="2026-09-09 20:46:09.363006668 +0000 UTC m=+2094.398477827"
Sep 09 20:46:09.364473 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:09.364444    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:18.364737 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:18.364697    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:18.376710 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:18.376680    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:18.403316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:18.403290    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:46:45.699497 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:45.699455    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"]
Sep 09 20:46:45.709012 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:46:45.708965    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:46:45.709012 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:46:45.709012 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:46:45.709012 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44" cmd=["/bin/sleep","15"]
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:45.709036    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" message="" err=<
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:46:45.709052    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="main" containerID="cri-o://a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44"
Sep 09 20:46:45.709271 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:45.709116    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="main" containerID="cri-o://a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44" gracePeriod=60
Sep 09 20:46:45.966526 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:45.966428    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 09 20:46:46.066859 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.066831    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:46:46.084143 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m84g\" (UniqueName: \"kubernetes.io/projected/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kube-api-access-9m84g\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kserve-provision-location\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084191    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-home\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084217    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-model-cache\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084247    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-dshm\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084265    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tls-certs\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084556 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084308    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tmp-dir\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084556 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084329    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ed047-63e9-4855-ad2b-685e06e90c7a-cabundle-cert\") pod \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\" (UID: \"dc2ed047-63e9-4855-ad2b-685e06e90c7a\") "
Sep 09 20:46:46.084556 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.084556 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084491    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-home" (OuterVolumeSpecName: "home") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.084769 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084655    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.084769 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084673    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.084769 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084649    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.085059 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.084794    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2ed047-63e9-4855-ad2b-685e06e90c7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:46.086727 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.086694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-dshm" (OuterVolumeSpecName: "dshm") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.086874 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.086854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kube-api-access-9m84g" (OuterVolumeSpecName: "kube-api-access-9m84g") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "kube-api-access-9m84g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:46.086935 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.086862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:46.108803 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.108760    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc2ed047-63e9-4855-ad2b-685e06e90c7a" (UID: "dc2ed047-63e9-4855-ad2b-685e06e90c7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:46.185268 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.185228    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.185268 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.185263    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.185268 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.185276    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.185526 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.185288    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ed047-63e9-4855-ad2b-685e06e90c7a-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.185526 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.185299    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m84g\" (UniqueName: \"kubernetes.io/projected/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kube-api-access-9m84g\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.185526 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.185312    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc2ed047-63e9-4855-ad2b-685e06e90c7a-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:46:46.527579 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.527547    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerID="a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44" exitCode=0
Sep 09 20:46:46.527579 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.527584    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" event={"ID":"dc2ed047-63e9-4855-ad2b-685e06e90c7a","Type":"ContainerDied","Data":"a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44"}
Sep 09 20:46:46.527804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.527606    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"
Sep 09 20:46:46.527804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.527621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22" event={"ID":"dc2ed047-63e9-4855-ad2b-685e06e90c7a","Type":"ContainerDied","Data":"640a8f69d648a2c91f6afdfd45903071c073b7422682cc2726907fcd3c51549d"}
Sep 09 20:46:46.527804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.527641    2566 scope.go:117] "RemoveContainer" containerID="a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44"
Sep 09 20:46:46.537782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.537754    2566 scope.go:117] "RemoveContainer" containerID="9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc"
Sep 09 20:46:46.545965 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.545946    2566 scope.go:117] "RemoveContainer" containerID="a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44"
Sep 09 20:46:46.546261 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:46:46.546242    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44\": container with ID starting with a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44 not found: ID does not exist" containerID="a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44"
Sep 09 20:46:46.546316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.546269    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44"} err="failed to get container status \"a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44\": rpc error: code = NotFound desc = could not find container \"a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44\": container with ID starting with a45bbcbe00aa3fe16609d9210d0fa31b1f3a25faa27cb5932acb808c6331cf44 not found: ID does not exist"
Sep 09 20:46:46.546316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.546290    2566 scope.go:117] "RemoveContainer" containerID="9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc"
Sep 09 20:46:46.546571 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:46:46.546549    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc\": container with ID starting with 9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc not found: ID does not exist" containerID="9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc"
Sep 09 20:46:46.546664 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.546578    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc"} err="failed to get container status \"9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc\": rpc error: code = NotFound desc = could not find container \"9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc\": container with ID starting with 9e56575dffdc762a56d61714f3a07718692663a4b5ba7c2eebb8b4d283ea12bc not found: ID does not exist"
Sep 09 20:46:46.569387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.569354    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"]
Sep 09 20:46:46.572386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:46.571964    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9b65d76bd-6jp22"]
Sep 09 20:46:47.491789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:47.491748    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" path="/var/lib/kubelet/pods/dc2ed047-63e9-4855-ad2b-685e06e90c7a/volumes"
Sep 09 20:46:58.664749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:46:58.664713    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"]
Sep 09 20:47:13.685039 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:13.684922    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9" podUID="5da45b73-8860-4d58-9e5e-be2774f7aab0" containerName="main" containerID="cri-o://c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0" gracePeriod=45
Sep 09 20:47:13.937300 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:13.937244    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:47:14.027718 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.027688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da45b73-8860-4d58-9e5e-be2774f7aab0-tls-certs\") pod \"5da45b73-8860-4d58-9e5e-be2774f7aab0\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") "
Sep 09 20:47:14.027902 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.027736    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-home\") pod \"5da45b73-8860-4d58-9e5e-be2774f7aab0\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") "
Sep 09 20:47:14.027902 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.027755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9pg2\" (UniqueName: \"kubernetes.io/projected/5da45b73-8860-4d58-9e5e-be2774f7aab0-kube-api-access-x9pg2\") pod \"5da45b73-8860-4d58-9e5e-be2774f7aab0\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") "
Sep 09 20:47:14.027902 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.027792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-tmp-dir\") pod \"5da45b73-8860-4d58-9e5e-be2774f7aab0\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") "
Sep 09 20:47:14.027902 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.027855    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-dshm\") pod \"5da45b73-8860-4d58-9e5e-be2774f7aab0\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") "
Sep 09 20:47:14.028169 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.027987    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-model-cache\") pod \"5da45b73-8860-4d58-9e5e-be2774f7aab0\" (UID: \"5da45b73-8860-4d58-9e5e-be2774f7aab0\") "
Sep 09 20:47:14.028169 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.028004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-home" (OuterVolumeSpecName: "home") pod "5da45b73-8860-4d58-9e5e-be2774f7aab0" (UID: "5da45b73-8860-4d58-9e5e-be2774f7aab0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:14.028169 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.028151    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5da45b73-8860-4d58-9e5e-be2774f7aab0" (UID: "5da45b73-8860-4d58-9e5e-be2774f7aab0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:14.028311 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.028257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-model-cache" (OuterVolumeSpecName: "model-cache") pod "5da45b73-8860-4d58-9e5e-be2774f7aab0" (UID: "5da45b73-8860-4d58-9e5e-be2774f7aab0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:14.028398 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.028380    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:14.028441 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.028402    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:14.028441 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.028412    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:14.030144 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.030127    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da45b73-8860-4d58-9e5e-be2774f7aab0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5da45b73-8860-4d58-9e5e-be2774f7aab0" (UID: "5da45b73-8860-4d58-9e5e-be2774f7aab0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:14.030495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.030472    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-dshm" (OuterVolumeSpecName: "dshm") pod "5da45b73-8860-4d58-9e5e-be2774f7aab0" (UID: "5da45b73-8860-4d58-9e5e-be2774f7aab0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:14.030594 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.030499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da45b73-8860-4d58-9e5e-be2774f7aab0-kube-api-access-x9pg2" (OuterVolumeSpecName: "kube-api-access-x9pg2") pod "5da45b73-8860-4d58-9e5e-be2774f7aab0" (UID: "5da45b73-8860-4d58-9e5e-be2774f7aab0"). InnerVolumeSpecName "kube-api-access-x9pg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:14.128964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.128926    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da45b73-8860-4d58-9e5e-be2774f7aab0-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:14.128964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.128961    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da45b73-8860-4d58-9e5e-be2774f7aab0-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:14.128964 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.128974    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9pg2\" (UniqueName: \"kubernetes.io/projected/5da45b73-8860-4d58-9e5e-be2774f7aab0-kube-api-access-x9pg2\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:14.666230 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.666189    2566 generic.go:356] "Generic (PLEG): container finished" podID="5da45b73-8860-4d58-9e5e-be2774f7aab0" containerID="c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0" exitCode=0
Sep 09 20:47:14.666421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.666233    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"
Sep 09 20:47:14.666421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.666257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9" event={"ID":"5da45b73-8860-4d58-9e5e-be2774f7aab0","Type":"ContainerDied","Data":"c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0"}
Sep 09 20:47:14.666421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.666299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9" event={"ID":"5da45b73-8860-4d58-9e5e-be2774f7aab0","Type":"ContainerDied","Data":"aa5e725804b479e725c6bcfd8db4b6b93e4fac68437db2863a85d28cb8bc164c"}
Sep 09 20:47:14.666421 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.666320    2566 scope.go:117] "RemoveContainer" containerID="c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0"
Sep 09 20:47:14.681232 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.681211    2566 scope.go:117] "RemoveContainer" containerID="c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0"
Sep 09 20:47:14.681562 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:47:14.681540    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0\": container with ID starting with c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0 not found: ID does not exist" containerID="c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0"
Sep 09 20:47:14.681612 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.681570    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0"} err="failed to get container status \"c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0\": rpc error: code = NotFound desc = could not find container \"c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0\": container with ID starting with c0775cacbd84988a65e0bc3245b839f94d67ebf182c71d57fd78bdacd95f39d0 not found: ID does not exist"
Sep 09 20:47:14.692612 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.692588    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"]
Sep 09 20:47:14.696831 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:14.696808    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-hhgd9"]
Sep 09 20:47:15.251811 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:15.251775    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"]
Sep 09 20:47:15.492948 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:15.492910    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da45b73-8860-4d58-9e5e-be2774f7aab0" path="/var/lib/kubelet/pods/5da45b73-8860-4d58-9e5e-be2774f7aab0/volumes"
Sep 09 20:47:21.257625 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.257587    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"]
Sep 09 20:47:21.258015 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.257974    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="storage-initializer"
Sep 09 20:47:21.258015 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.257992    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="storage-initializer"
Sep 09 20:47:21.258102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.258023    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="main"
Sep 09 20:47:21.258102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.258033    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="main"
Sep 09 20:47:21.258102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.258041    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da45b73-8860-4d58-9e5e-be2774f7aab0" containerName="main"
Sep 09 20:47:21.258102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.258046    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da45b73-8860-4d58-9e5e-be2774f7aab0" containerName="main"
Sep 09 20:47:21.258234 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.258130    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc2ed047-63e9-4855-ad2b-685e06e90c7a" containerName="main"
Sep 09 20:47:21.258234 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.258139    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da45b73-8860-4d58-9e5e-be2774f7aab0" containerName="main"
Sep 09 20:47:21.261053 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.261025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.263608 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.263585    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-9jprl\""
Sep 09 20:47:21.263740 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.263639    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 09 20:47:21.264708 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.264690    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 09 20:47:21.264984 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.264964    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 09 20:47:21.271246 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.271221    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"]
Sep 09 20:47:21.398168 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.398132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrz84\" (UniqueName: \"kubernetes.io/projected/e91f91bb-6800-463c-959a-52016b25fce7-kube-api-access-jrz84\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.398168 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.398170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-home\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.398393 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.398212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-model-cache\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.398393 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.398232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91f91bb-6800-463c-959a-52016b25fce7-tls-certs\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.398393 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.398252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-dshm\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.398393 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.398323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499231 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz84\" (UniqueName: \"kubernetes.io/projected/e91f91bb-6800-463c-959a-52016b25fce7-kube-api-access-jrz84\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499231 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-home\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499464 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-model-cache\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499464 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91f91bb-6800-463c-959a-52016b25fce7-tls-certs\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499464 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-dshm\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499464 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-home\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499736 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.499794 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.499755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-model-cache\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.501764 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.501745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-dshm\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.502019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.502000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91f91bb-6800-463c-959a-52016b25fce7-tls-certs\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.508651 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.508599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz84\" (UniqueName: \"kubernetes.io/projected/e91f91bb-6800-463c-959a-52016b25fce7-kube-api-access-jrz84\") pod \"leave-v1-kserve-65684d4f5-bc2dh\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.573163 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.573128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:21.855684 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:21.855655    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"]
Sep 09 20:47:21.858162 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:47:21.858127    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91f91bb_6800_463c_959a_52016b25fce7.slice/crio-085263ba4f4bafa71436320fb4af03b686afcc266065a4f03fb74a2f8866d867 WatchSource:0}: Error finding container 085263ba4f4bafa71436320fb4af03b686afcc266065a4f03fb74a2f8866d867: Status 404 returned error can't find the container with id 085263ba4f4bafa71436320fb4af03b686afcc266065a4f03fb74a2f8866d867
Sep 09 20:47:22.708265 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:22.708226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh" event={"ID":"e91f91bb-6800-463c-959a-52016b25fce7","Type":"ContainerStarted","Data":"c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1"}
Sep 09 20:47:22.708648 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:22.708279    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh" event={"ID":"e91f91bb-6800-463c-959a-52016b25fce7","Type":"ContainerStarted","Data":"085263ba4f4bafa71436320fb4af03b686afcc266065a4f03fb74a2f8866d867"}
Sep 09 20:47:22.727766 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:22.727714    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh" podStartSLOduration=1.72769828 podStartE2EDuration="1.72769828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:22.725910262 +0000 UTC m=+2167.761381434" watchObservedRunningTime="2026-09-09 20:47:22.72769828 +0000 UTC m=+2167.763169441"
Sep 09 20:47:23.573913 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:23.573874    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:30.274372 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.274329    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll" podUID="7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" containerName="main" containerID="cri-o://843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c" gracePeriod=45
Sep 09 20:47:30.305625 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.305589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll" podUID="7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 09 20:47:30.355622 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.355586    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"]
Sep 09 20:47:30.359098 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.359063    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.361769 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.361750    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 09 20:47:30.371842 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.371813    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"]
Sep 09 20:47:30.375271 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.375247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-home\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.375368 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.375352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ptnz\" (UniqueName: \"kubernetes.io/projected/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-kube-api-access-6ptnz\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.375423 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.375410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-model-cache\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.375464 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.375432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.375464 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.375455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-dshm\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.375561 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.375481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.476441 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ptnz\" (UniqueName: \"kubernetes.io/projected/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-kube-api-access-6ptnz\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.476592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-model-cache\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.476592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.476592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-dshm\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.476592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.476592 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-home\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.476975 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-model-cache\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.477072 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.476990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.477072 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.477006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-home\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.479179 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.479157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-dshm\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.479301 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.479282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.485888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.485857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ptnz\" (UniqueName: \"kubernetes.io/projected/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-kube-api-access-6ptnz\") pod \"leave-v2-kserve-556b4dddf9-fpm8v\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.528715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.528652    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:47:30.577482 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577450    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-model-cache\") pod \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") "
Sep 09 20:47:30.577650 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577488    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xrp\" (UniqueName: \"kubernetes.io/projected/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-kube-api-access-m8xrp\") pod \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") "
Sep 09 20:47:30.577650 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-home\") pod \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") "
Sep 09 20:47:30.577650 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577541    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-dshm\") pod \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") "
Sep 09 20:47:30.577650 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577590    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tmp-dir\") pod \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") "
Sep 09 20:47:30.577650 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577619    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tls-certs\") pod \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\" (UID: \"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc\") "
Sep 09 20:47:30.577928 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-model-cache" (OuterVolumeSpecName: "model-cache") pod "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" (UID: "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:30.577928 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" (UID: "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:30.577928 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-home" (OuterVolumeSpecName: "home") pod "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" (UID: "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:30.577928 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577914    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:30.578116 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577933    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:30.578116 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.577947    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:30.579854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.579830    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-kube-api-access-m8xrp" (OuterVolumeSpecName: "kube-api-access-m8xrp") pod "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" (UID: "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc"). InnerVolumeSpecName "kube-api-access-m8xrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:30.579854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.579837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-dshm" (OuterVolumeSpecName: "dshm") pod "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" (UID: "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:30.579854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.579842    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" (UID: "7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:30.670159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.670127    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:30.678433 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.678415    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:30.678517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.678437    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xrp\" (UniqueName: \"kubernetes.io/projected/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-kube-api-access-m8xrp\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:30.678517 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.678446    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:47:30.747749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.747719    2566 generic.go:356] "Generic (PLEG): container finished" podID="7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" containerID="843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c" exitCode=0
Sep 09 20:47:30.747884 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.747773    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"
Sep 09 20:47:30.747884 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.747809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll" event={"ID":"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc","Type":"ContainerDied","Data":"843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c"}
Sep 09 20:47:30.747975 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.747882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll" event={"ID":"7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc","Type":"ContainerDied","Data":"a4658805f3396e96c8ed5c1b373ed8fe1f6c60675ae9bb9ea01dbd4cd5f60bde"}
Sep 09 20:47:30.747975 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.747918    2566 scope.go:117] "RemoveContainer" containerID="843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c"
Sep 09 20:47:30.757348 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.757329    2566 scope.go:117] "RemoveContainer" containerID="843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c"
Sep 09 20:47:30.757664 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:47:30.757643    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c\": container with ID starting with 843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c not found: ID does not exist" containerID="843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c"
Sep 09 20:47:30.757742 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.757675    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c"} err="failed to get container status \"843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c\": rpc error: code = NotFound desc = could not find container \"843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c\": container with ID starting with 843d1e07196fcfddc7d2457a069dcc943fa9f3b5dcc573ad4361e272a6bdbe5c not found: ID does not exist"
Sep 09 20:47:30.771602 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.771572    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"]
Sep 09 20:47:30.777367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.777342    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-sx2ll"]
Sep 09 20:47:30.945371 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:30.945336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"]
Sep 09 20:47:30.948730 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:47:30.948689    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d7f44f_16bb_41fb_b688_b4fbb66a0a08.slice/crio-0d47d673847bdb20bc52eb2d388ddd4c52c85e4e44b7ebf7db0e8edda7ac5f70 WatchSource:0}: Error finding container 0d47d673847bdb20bc52eb2d388ddd4c52c85e4e44b7ebf7db0e8edda7ac5f70: Status 404 returned error can't find the container with id 0d47d673847bdb20bc52eb2d388ddd4c52c85e4e44b7ebf7db0e8edda7ac5f70
Sep 09 20:47:31.492589 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:31.492547    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" path="/var/lib/kubelet/pods/7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc/volumes"
Sep 09 20:47:31.574410 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:31.574332    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:31.587102 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:31.587054    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:31.761523 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:31.761483    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v" event={"ID":"37d7f44f-16bb-41fb-b688-b4fbb66a0a08","Type":"ContainerStarted","Data":"b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d"}
Sep 09 20:47:31.761724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:31.761557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v" event={"ID":"37d7f44f-16bb-41fb-b688-b4fbb66a0a08","Type":"ContainerStarted","Data":"0d47d673847bdb20bc52eb2d388ddd4c52c85e4e44b7ebf7db0e8edda7ac5f70"}
Sep 09 20:47:31.773686 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:31.773657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:47:31.788442 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:31.788400    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v" podStartSLOduration=1.7883864059999999 podStartE2EDuration="1.788386406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:31.785833786 +0000 UTC m=+2176.821304945" watchObservedRunningTime="2026-09-09 20:47:31.788386406 +0000 UTC m=+2176.823857583"
Sep 09 20:47:32.671252 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:32.671216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:40.670424 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:40.670382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:40.682806 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:40.682772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:40.816409 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:40.816382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:47:56.581096 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.581055    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"]
Sep 09 20:47:56.581531 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.581457    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" containerName="main"
Sep 09 20:47:56.581531 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.581470    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" containerName="main"
Sep 09 20:47:56.581605 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.581533    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7185dc82-3fbe-4658-bbc7-4fb1b2b7bddc" containerName="main"
Sep 09 20:47:56.584556 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.584540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.587552 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.587527    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 09 20:47:56.587709 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.587661    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 09 20:47:56.588580 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.588564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:47:56.588707 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.588610    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 09 20:47:56.588707 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.588680    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-nfjpv\""
Sep 09 20:47:56.588884 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.588702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 09 20:47:56.594262 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.594241    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"]
Sep 09 20:47:56.696336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-home\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.696336 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.696618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.696618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qx7\" (UniqueName: \"kubernetes.io/projected/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kube-api-access-24qx7\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.696618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-dshm\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.696618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.696618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.696618 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.696617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34185ff8-e6a0-4d8e-94d9-def3d143fca2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.797804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.797769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-dshm\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.797804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.797806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.797831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.797860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34185ff8-e6a0-4d8e-94d9-def3d143fca2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.797919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-home\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.797950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.797983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.798004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qx7\" (UniqueName: \"kubernetes.io/projected/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kube-api-access-24qx7\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798360 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.798323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798437 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.798412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798792 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.798774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-home\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798894 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.798870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34185ff8-e6a0-4d8e-94d9-def3d143fca2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.798933 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.798903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.800319 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.800299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-dshm\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.800866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.800846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.806397 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.806377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qx7\" (UniqueName: \"kubernetes.io/projected/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kube-api-access-24qx7\") pod \"tokenizer-migration-path-test-kserve-7b7c969f7-n29h5\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:56.895996 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:56.895965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:47:57.173667 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:57.173590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"]
Sep 09 20:47:57.177130 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:47:57.177097    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34185ff8_e6a0_4d8e_94d9_def3d143fca2.slice/crio-cce6dabee4ef73a38f06f765e123adac1a6b95e64ef49ca94407a9521fcf2ff0 WatchSource:0}: Error finding container cce6dabee4ef73a38f06f765e123adac1a6b95e64ef49ca94407a9521fcf2ff0: Status 404 returned error can't find the container with id cce6dabee4ef73a38f06f765e123adac1a6b95e64ef49ca94407a9521fcf2ff0
Sep 09 20:47:57.893020 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:57.892972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" event={"ID":"34185ff8-e6a0-4d8e-94d9-def3d143fca2","Type":"ContainerStarted","Data":"e98d82a1b7a4145f64a7bdbf9ab5c2b4ce1e58b7fcf60eb7bcd9abf4d0bb93c5"}
Sep 09 20:47:57.893429 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:57.893047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" event={"ID":"34185ff8-e6a0-4d8e-94d9-def3d143fca2","Type":"ContainerStarted","Data":"cce6dabee4ef73a38f06f765e123adac1a6b95e64ef49ca94407a9521fcf2ff0"}
Sep 09 20:47:59.905608 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:59.905570    2566 generic.go:356] "Generic (PLEG): container finished" podID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerID="e98d82a1b7a4145f64a7bdbf9ab5c2b4ce1e58b7fcf60eb7bcd9abf4d0bb93c5" exitCode=0
Sep 09 20:47:59.905989 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:47:59.905644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" event={"ID":"34185ff8-e6a0-4d8e-94d9-def3d143fca2","Type":"ContainerDied","Data":"e98d82a1b7a4145f64a7bdbf9ab5c2b4ce1e58b7fcf60eb7bcd9abf4d0bb93c5"}
Sep 09 20:48:00.912766 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:00.912730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" event={"ID":"34185ff8-e6a0-4d8e-94d9-def3d143fca2","Type":"ContainerStarted","Data":"82c37e17fa8adf2be411dd5a57aa8ad0d855e64bee66e94906dccebfcea54d2f"}
Sep 09 20:48:00.934703 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:00.934647    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" podStartSLOduration=2.644165197 podStartE2EDuration="4.934630735s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.290465538s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:00.931761971 +0000 UTC m=+2205.967233130" watchObservedRunningTime="2026-09-09 20:48:00.934630735 +0000 UTC m=+2205.970101895"
Sep 09 20:48:01.896246 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:01.896216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:48:06.896294 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:06.896254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:48:06.909127 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:06.909065    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:48:06.956862 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:06.956833    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:48:14.139170 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:14.139134    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"]
Sep 09 20:48:14.944386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:14.944352    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"]
Sep 09 20:48:23.066657 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.066617    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"]
Sep 09 20:48:23.070289 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.070266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.073908 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.073876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-r6rr9\""
Sep 09 20:48:23.073908 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.073887    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 09 20:48:23.074070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.073876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 09 20:48:23.074070 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.073930    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 09 20:48:23.088255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.088193    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"]
Sep 09 20:48:23.132984 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.132947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-dshm\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.133213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.133186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.133376 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.133359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fkjm\" (UniqueName: \"kubernetes.io/projected/c11c6084-7cfe-4c79-b643-4604ac40ebd0-kube-api-access-8fkjm\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.133514 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.133499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-home\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.133633 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.133620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.133797 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.133781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.234634 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.234602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-dshm\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.234634 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.234638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.234867 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.234676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fkjm\" (UniqueName: \"kubernetes.io/projected/c11c6084-7cfe-4c79-b643-4604ac40ebd0-kube-api-access-8fkjm\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.234867 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.234709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-home\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.234867 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.234745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.234867 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.234808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.235145 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.235121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.235218 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.235174    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-home\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.235218 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.235185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.237007 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.236984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-dshm\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.237316 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.237299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.243681 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.243660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fkjm\" (UniqueName: \"kubernetes.io/projected/c11c6084-7cfe-4c79-b643-4604ac40ebd0-kube-api-access-8fkjm\") pod \"tri-v1-kserve-79f8f98bc5-zgmt5\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.380228 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.380188    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:23.656175 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:23.656142    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"]
Sep 09 20:48:23.657720 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:48:23.657690    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11c6084_7cfe_4c79_b643_4604ac40ebd0.slice/crio-52bbc424be6c9709aee37bc7800207df839f2545ad703b143c842159ae22f724 WatchSource:0}: Error finding container 52bbc424be6c9709aee37bc7800207df839f2545ad703b143c842159ae22f724: Status 404 returned error can't find the container with id 52bbc424be6c9709aee37bc7800207df839f2545ad703b143c842159ae22f724
Sep 09 20:48:24.028797 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:24.028707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5" event={"ID":"c11c6084-7cfe-4c79-b643-4604ac40ebd0","Type":"ContainerStarted","Data":"8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19"}
Sep 09 20:48:24.028797 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:24.028784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5" event={"ID":"c11c6084-7cfe-4c79-b643-4604ac40ebd0","Type":"ContainerStarted","Data":"52bbc424be6c9709aee37bc7800207df839f2545ad703b143c842159ae22f724"}
Sep 09 20:48:24.046163 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:24.046107    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5" podStartSLOduration=1.046065031 podStartE2EDuration="1.046065031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:24.045783696 +0000 UTC m=+2229.081254856" watchObservedRunningTime="2026-09-09 20:48:24.046065031 +0000 UTC m=+2229.081536191"
Sep 09 20:48:24.380581 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:24.380539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:29.160949 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.160887    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh" podUID="e91f91bb-6800-463c-959a-52016b25fce7" containerName="main" containerID="cri-o://c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1" gracePeriod=45
Sep 09 20:48:29.411025 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.410959    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:48:29.488200 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488173    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrz84\" (UniqueName: \"kubernetes.io/projected/e91f91bb-6800-463c-959a-52016b25fce7-kube-api-access-jrz84\") pod \"e91f91bb-6800-463c-959a-52016b25fce7\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") "
Sep 09 20:48:29.488378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488227    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-tmp-dir\") pod \"e91f91bb-6800-463c-959a-52016b25fce7\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") "
Sep 09 20:48:29.488378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-model-cache\") pod \"e91f91bb-6800-463c-959a-52016b25fce7\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") "
Sep 09 20:48:29.488378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488338    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-home\") pod \"e91f91bb-6800-463c-959a-52016b25fce7\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") "
Sep 09 20:48:29.488540 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91f91bb-6800-463c-959a-52016b25fce7-tls-certs\") pod \"e91f91bb-6800-463c-959a-52016b25fce7\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") "
Sep 09 20:48:29.488540 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-dshm\") pod \"e91f91bb-6800-463c-959a-52016b25fce7\" (UID: \"e91f91bb-6800-463c-959a-52016b25fce7\") "
Sep 09 20:48:29.488540 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e91f91bb-6800-463c-959a-52016b25fce7" (UID: "e91f91bb-6800-463c-959a-52016b25fce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:29.488694 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-home" (OuterVolumeSpecName: "home") pod "e91f91bb-6800-463c-959a-52016b25fce7" (UID: "e91f91bb-6800-463c-959a-52016b25fce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:29.488754 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488741    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:29.488813 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488760    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:29.488813 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.488794    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e91f91bb-6800-463c-959a-52016b25fce7" (UID: "e91f91bb-6800-463c-959a-52016b25fce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:29.490694 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.490662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-dshm" (OuterVolumeSpecName: "dshm") pod "e91f91bb-6800-463c-959a-52016b25fce7" (UID: "e91f91bb-6800-463c-959a-52016b25fce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:29.490694 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.490675    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91f91bb-6800-463c-959a-52016b25fce7-kube-api-access-jrz84" (OuterVolumeSpecName: "kube-api-access-jrz84") pod "e91f91bb-6800-463c-959a-52016b25fce7" (UID: "e91f91bb-6800-463c-959a-52016b25fce7"). InnerVolumeSpecName "kube-api-access-jrz84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:29.490841 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.490738    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91f91bb-6800-463c-959a-52016b25fce7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e91f91bb-6800-463c-959a-52016b25fce7" (UID: "e91f91bb-6800-463c-959a-52016b25fce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:48:29.589509 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.589477    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91f91bb-6800-463c-959a-52016b25fce7-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:29.589509 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.589503    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:29.589686 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.589517    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrz84\" (UniqueName: \"kubernetes.io/projected/e91f91bb-6800-463c-959a-52016b25fce7-kube-api-access-jrz84\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:29.589686 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.589529    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91f91bb-6800-463c-959a-52016b25fce7-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:29.965613 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:29.965555    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v" podUID="37d7f44f-16bb-41fb-b688-b4fbb66a0a08" containerName="main" containerID="cri-o://b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d" gracePeriod=45
Sep 09 20:48:30.060241 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.060209    2566 generic.go:356] "Generic (PLEG): container finished" podID="e91f91bb-6800-463c-959a-52016b25fce7" containerID="c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1" exitCode=0
Sep 09 20:48:30.060417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.060251    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"
Sep 09 20:48:30.060417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.060287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh" event={"ID":"e91f91bb-6800-463c-959a-52016b25fce7","Type":"ContainerDied","Data":"c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1"}
Sep 09 20:48:30.060417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.060339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh" event={"ID":"e91f91bb-6800-463c-959a-52016b25fce7","Type":"ContainerDied","Data":"085263ba4f4bafa71436320fb4af03b686afcc266065a4f03fb74a2f8866d867"}
Sep 09 20:48:30.060417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.060363    2566 scope.go:117] "RemoveContainer" containerID="c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1"
Sep 09 20:48:30.077631 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.077607    2566 scope.go:117] "RemoveContainer" containerID="c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1"
Sep 09 20:48:30.078007 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:48:30.077987    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1\": container with ID starting with c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1 not found: ID does not exist" containerID="c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1"
Sep 09 20:48:30.078145 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.078021    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1"} err="failed to get container status \"c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1\": rpc error: code = NotFound desc = could not find container \"c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1\": container with ID starting with c5d2accb4c2ea0ae4e228f38ab2a1babca3c07a3cd26636a67d2da8a8429bff1 not found: ID does not exist"
Sep 09 20:48:30.078385 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.078363    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"]
Sep 09 20:48:30.082988 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.082959    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-bc2dh"]
Sep 09 20:48:30.230513 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.230456    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:48:30.295883 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.295843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-home\") pod \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") "
Sep 09 20:48:30.295883 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.295882    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ptnz\" (UniqueName: \"kubernetes.io/projected/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-kube-api-access-6ptnz\") pod \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") "
Sep 09 20:48:30.296159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.295917    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tmp-dir\") pod \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") "
Sep 09 20:48:30.296159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.295933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-dshm\") pod \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") "
Sep 09 20:48:30.296159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.295964    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-model-cache\") pod \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") "
Sep 09 20:48:30.296159 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.295990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tls-certs\") pod \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\" (UID: \"37d7f44f-16bb-41fb-b688-b4fbb66a0a08\") "
Sep 09 20:48:30.296377 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.296167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-home" (OuterVolumeSpecName: "home") pod "37d7f44f-16bb-41fb-b688-b4fbb66a0a08" (UID: "37d7f44f-16bb-41fb-b688-b4fbb66a0a08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:30.296377 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.296184    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37d7f44f-16bb-41fb-b688-b4fbb66a0a08" (UID: "37d7f44f-16bb-41fb-b688-b4fbb66a0a08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:30.296377 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.296293    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:30.296377 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.296293    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-model-cache" (OuterVolumeSpecName: "model-cache") pod "37d7f44f-16bb-41fb-b688-b4fbb66a0a08" (UID: "37d7f44f-16bb-41fb-b688-b4fbb66a0a08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:30.296377 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.296307    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:30.298474 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.298452    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-kube-api-access-6ptnz" (OuterVolumeSpecName: "kube-api-access-6ptnz") pod "37d7f44f-16bb-41fb-b688-b4fbb66a0a08" (UID: "37d7f44f-16bb-41fb-b688-b4fbb66a0a08"). InnerVolumeSpecName "kube-api-access-6ptnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:30.298577 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.298449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-dshm" (OuterVolumeSpecName: "dshm") pod "37d7f44f-16bb-41fb-b688-b4fbb66a0a08" (UID: "37d7f44f-16bb-41fb-b688-b4fbb66a0a08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:30.298577 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.298450    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37d7f44f-16bb-41fb-b688-b4fbb66a0a08" (UID: "37d7f44f-16bb-41fb-b688-b4fbb66a0a08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:48:30.397207 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.397166    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:30.397207 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.397199    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:30.397207 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.397214    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:30.397460 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:30.397227    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ptnz\" (UniqueName: \"kubernetes.io/projected/37d7f44f-16bb-41fb-b688-b4fbb66a0a08-kube-api-access-6ptnz\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:48:31.068353 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.068316    2566 generic.go:356] "Generic (PLEG): container finished" podID="37d7f44f-16bb-41fb-b688-b4fbb66a0a08" containerID="b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d" exitCode=0
Sep 09 20:48:31.068562 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.068366    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"
Sep 09 20:48:31.068562 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.068403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v" event={"ID":"37d7f44f-16bb-41fb-b688-b4fbb66a0a08","Type":"ContainerDied","Data":"b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d"}
Sep 09 20:48:31.068562 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.068463    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v" event={"ID":"37d7f44f-16bb-41fb-b688-b4fbb66a0a08","Type":"ContainerDied","Data":"0d47d673847bdb20bc52eb2d388ddd4c52c85e4e44b7ebf7db0e8edda7ac5f70"}
Sep 09 20:48:31.068562 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.068490    2566 scope.go:117] "RemoveContainer" containerID="b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d"
Sep 09 20:48:31.077715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.077694    2566 scope.go:117] "RemoveContainer" containerID="b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d"
Sep 09 20:48:31.078053 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:48:31.078022    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d\": container with ID starting with b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d not found: ID does not exist" containerID="b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d"
Sep 09 20:48:31.078178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.078063    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d"} err="failed to get container status \"b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d\": rpc error: code = NotFound desc = could not find container \"b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d\": container with ID starting with b09c8e0af5f7ecd202f7287e4c4ab387fc9330a408d782f474e041908f55e02d not found: ID does not exist"
Sep 09 20:48:31.091484 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.091449    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"]
Sep 09 20:48:31.095693 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.095662    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-fpm8v"]
Sep 09 20:48:31.428318 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.428253    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"]
Sep 09 20:48:31.428844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.428819    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91f91bb-6800-463c-959a-52016b25fce7" containerName="main"
Sep 09 20:48:31.428844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.428846    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91f91bb-6800-463c-959a-52016b25fce7" containerName="main"
Sep 09 20:48:31.428995 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.428886    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d7f44f-16bb-41fb-b688-b4fbb66a0a08" containerName="main"
Sep 09 20:48:31.428995 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.428893    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d7f44f-16bb-41fb-b688-b4fbb66a0a08" containerName="main"
Sep 09 20:48:31.428995 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.428954    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d7f44f-16bb-41fb-b688-b4fbb66a0a08" containerName="main"
Sep 09 20:48:31.428995 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.428961    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e91f91bb-6800-463c-959a-52016b25fce7" containerName="main"
Sep 09 20:48:31.433266 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.433246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.435871 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.435850    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 09 20:48:31.447800 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.447774    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"]
Sep 09 20:48:31.493032 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.492993    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d7f44f-16bb-41fb-b688-b4fbb66a0a08" path="/var/lib/kubelet/pods/37d7f44f-16bb-41fb-b688-b4fbb66a0a08/volumes"
Sep 09 20:48:31.493427 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.493413    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e91f91bb-6800-463c-959a-52016b25fce7" path="/var/lib/kubelet/pods/e91f91bb-6800-463c-959a-52016b25fce7/volumes"
Sep 09 20:48:31.506454 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.506428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.506534 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.506475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-dshm\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.506534 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.506519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-home\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.506599 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.506579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91934607-2845-408b-9807-f60c80bff911-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.506636 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.506604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-model-cache\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.506670 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.506650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5zb\" (UniqueName: \"kubernetes.io/projected/91934607-2845-408b-9807-f60c80bff911-kube-api-access-5d5zb\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.607534 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.607496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91934607-2845-408b-9807-f60c80bff911-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.607534 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.607538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-model-cache\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.607749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.607673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5zb\" (UniqueName: \"kubernetes.io/projected/91934607-2845-408b-9807-f60c80bff911-kube-api-access-5d5zb\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.607749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.607729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.607824 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.607771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-dshm\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.607824 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.607796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-home\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.607952 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.607930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-model-cache\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.608147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.608063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.608147 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.608138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-home\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.610138 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.610108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-dshm\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.610382 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.610364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91934607-2845-408b-9807-f60c80bff911-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.616486 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.616461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5zb\" (UniqueName: \"kubernetes.io/projected/91934607-2845-408b-9807-f60c80bff911-kube-api-access-5d5zb\") pod \"tri-v2-kserve-dcb554bf5-f8j8q\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:31.743789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:31.743690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:32.019289 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:32.019261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"]
Sep 09 20:48:32.021385 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:48:32.021350    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91934607_2845_408b_9807_f60c80bff911.slice/crio-96fc903ca9a8fcd675e8e08bc46cfb36123c43ac4a2033f1b3d3700094a61134 WatchSource:0}: Error finding container 96fc903ca9a8fcd675e8e08bc46cfb36123c43ac4a2033f1b3d3700094a61134: Status 404 returned error can't find the container with id 96fc903ca9a8fcd675e8e08bc46cfb36123c43ac4a2033f1b3d3700094a61134
Sep 09 20:48:32.076305 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:32.076270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q" event={"ID":"91934607-2845-408b-9807-f60c80bff911","Type":"ContainerStarted","Data":"96fc903ca9a8fcd675e8e08bc46cfb36123c43ac4a2033f1b3d3700094a61134"}
Sep 09 20:48:33.082661 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:33.082612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q" event={"ID":"91934607-2845-408b-9807-f60c80bff911","Type":"ContainerStarted","Data":"b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc"}
Sep 09 20:48:33.101239 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:33.101182    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q" podStartSLOduration=2.101166809 podStartE2EDuration="2.101166809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:33.099347673 +0000 UTC m=+2238.134818837" watchObservedRunningTime="2026-09-09 20:48:33.101166809 +0000 UTC m=+2238.136637969"
Sep 09 20:48:33.380729 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:33.380700    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:33.393686 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:33.393654    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:33.744044 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:33.743962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:34.101017 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:34.100898    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:48:41.744310 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:41.744212    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:41.756766 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:41.756737    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:42.135039 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:42.135006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:48:51.346162 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.346124    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"]
Sep 09 20:48:51.349664 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.349640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.352500 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.352468    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 09 20:48:51.359264 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.359239    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"]
Sep 09 20:48:51.480062 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.480025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-dshm\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.480312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.480071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-model-cache\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.480312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.480124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.480312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.480203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-home\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.480312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.480250    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcz64\" (UniqueName: \"kubernetes.io/projected/8139bf0b-c457-4fc3-9c1e-a37e58f09924-kube-api-access-vcz64\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.480312 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.480285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tls-certs\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581109 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-dshm\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-model-cache\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581285 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-home\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581436 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcz64\" (UniqueName: \"kubernetes.io/projected/8139bf0b-c457-4fc3-9c1e-a37e58f09924-kube-api-access-vcz64\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581436 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tls-certs\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581595 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-model-cache\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581690 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.581748 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.581700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-home\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.583575 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.583554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-dshm\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.583854 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.583838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tls-certs\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.592908 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.592882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcz64\" (UniqueName: \"kubernetes.io/projected/8139bf0b-c457-4fc3-9c1e-a37e58f09924-kube-api-access-vcz64\") pod \"tri-v3-kserve-798c88c66c-4s26r\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.662126 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.662067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:48:51.935520 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:51.935476    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"]
Sep 09 20:48:51.938437 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:48:51.938409    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8139bf0b_c457_4fc3_9c1e_a37e58f09924.slice/crio-1328661cc0bb96bd4e51535521d4f068a6bce6b6c1b11ff69ca091b24953fb39 WatchSource:0}: Error finding container 1328661cc0bb96bd4e51535521d4f068a6bce6b6c1b11ff69ca091b24953fb39: Status 404 returned error can't find the container with id 1328661cc0bb96bd4e51535521d4f068a6bce6b6c1b11ff69ca091b24953fb39
Sep 09 20:48:52.171985 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:52.171944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r" event={"ID":"8139bf0b-c457-4fc3-9c1e-a37e58f09924","Type":"ContainerStarted","Data":"983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200"}
Sep 09 20:48:52.172178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:52.172016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r" event={"ID":"8139bf0b-c457-4fc3-9c1e-a37e58f09924","Type":"ContainerStarted","Data":"1328661cc0bb96bd4e51535521d4f068a6bce6b6c1b11ff69ca091b24953fb39"}
Sep 09 20:48:52.190043 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:52.189916    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r" podStartSLOduration=1.189892745 podStartE2EDuration="1.189892745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:52.189199413 +0000 UTC m=+2257.224670575" watchObservedRunningTime="2026-09-09 20:48:52.189892745 +0000 UTC m=+2257.225363906"
Sep 09 20:48:52.663101 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:48:52.663044    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:49:01.662567 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:01.662530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:49:01.675008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:01.674981    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:49:02.228969 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:02.228938    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:49:29.095671 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.095634    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"]
Sep 09 20:49:29.105300 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:49:29.105259    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:49:29.105300 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:49:29.105300 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:49:29.105300 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="82c37e17fa8adf2be411dd5a57aa8ad0d855e64bee66e94906dccebfcea54d2f" cmd=["/bin/sleep","15"]
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.105308    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" message="" err=<
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:49:29.105320    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerName="main" containerID="cri-o://82c37e17fa8adf2be411dd5a57aa8ad0d855e64bee66e94906dccebfcea54d2f"
Sep 09 20:49:29.105637 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.105369    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerName="main" containerID="cri-o://82c37e17fa8adf2be411dd5a57aa8ad0d855e64bee66e94906dccebfcea54d2f" gracePeriod=60
Sep 09 20:49:29.349495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.349404    2566 generic.go:356] "Generic (PLEG): container finished" podID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerID="82c37e17fa8adf2be411dd5a57aa8ad0d855e64bee66e94906dccebfcea54d2f" exitCode=0
Sep 09 20:49:29.349495 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.349474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" event={"ID":"34185ff8-e6a0-4d8e-94d9-def3d143fca2","Type":"ContainerDied","Data":"82c37e17fa8adf2be411dd5a57aa8ad0d855e64bee66e94906dccebfcea54d2f"}
Sep 09 20:49:29.452133 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.452098    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:49:29.509762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509729    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-home\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.509762 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-dshm\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.509961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-model-cache\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.509961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24qx7\" (UniqueName: \"kubernetes.io/projected/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kube-api-access-24qx7\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.509961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509866    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34185ff8-e6a0-4d8e-94d9-def3d143fca2-cabundle-cert\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.509961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509897    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kserve-provision-location\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.509961 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tmp-dir\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.510254 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.509987    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tls-certs\") pod \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\" (UID: \"34185ff8-e6a0-4d8e-94d9-def3d143fca2\") "
Sep 09 20:49:29.510254 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.510009    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-home" (OuterVolumeSpecName: "home") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.510254 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.510019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-model-cache" (OuterVolumeSpecName: "model-cache") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.510389 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.510258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.510389 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.510303    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34185ff8-e6a0-4d8e-94d9-def3d143fca2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:29.510389 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.510320    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.510389 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.510338    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.512283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.512258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-dshm" (OuterVolumeSpecName: "dshm") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.512283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.512272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kube-api-access-24qx7" (OuterVolumeSpecName: "kube-api-access-24qx7") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "kube-api-access-24qx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:29.512503 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.512479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:29.531327 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.531296    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34185ff8-e6a0-4d8e-94d9-def3d143fca2" (UID: "34185ff8-e6a0-4d8e-94d9-def3d143fca2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:29.610982 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.610958    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.611156 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.610985    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24qx7\" (UniqueName: \"kubernetes.io/projected/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kube-api-access-24qx7\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.611156 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.610996    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34185ff8-e6a0-4d8e-94d9-def3d143fca2-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.611156 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.611006    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.611156 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.611015    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:29.611156 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:29.611025    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34185ff8-e6a0-4d8e-94d9-def3d143fca2-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:49:30.355914 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:30.355886    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"
Sep 09 20:49:30.356346 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:30.355886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5" event={"ID":"34185ff8-e6a0-4d8e-94d9-def3d143fca2","Type":"ContainerDied","Data":"cce6dabee4ef73a38f06f765e123adac1a6b95e64ef49ca94407a9521fcf2ff0"}
Sep 09 20:49:30.356346 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:30.356029    2566 scope.go:117] "RemoveContainer" containerID="82c37e17fa8adf2be411dd5a57aa8ad0d855e64bee66e94906dccebfcea54d2f"
Sep 09 20:49:30.365909 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:30.365888    2566 scope.go:117] "RemoveContainer" containerID="e98d82a1b7a4145f64a7bdbf9ab5c2b4ce1e58b7fcf60eb7bcd9abf4d0bb93c5"
Sep 09 20:49:30.380645 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:30.380620    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"]
Sep 09 20:49:30.384676 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:30.384654    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b7c969f7-n29h5"]
Sep 09 20:49:31.492019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:49:31.491978    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" path="/var/lib/kubelet/pods/34185ff8-e6a0-4d8e-94d9-def3d143fca2/volumes"
Sep 09 20:50:08.767740 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:08.767661    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"]
Sep 09 20:50:09.576614 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:09.576576    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"]
Sep 09 20:50:09.951883 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:09.951849    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"]
Sep 09 20:50:23.786927 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:23.786850    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r" podUID="8139bf0b-c457-4fc3-9c1e-a37e58f09924" containerName="main" containerID="cri-o://983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200" gracePeriod=45
Sep 09 20:50:24.033631 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.033604    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:50:24.083804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.083718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-home\") pod \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") "
Sep 09 20:50:24.083804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.083754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tls-certs\") pod \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") "
Sep 09 20:50:24.083804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.083778    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-model-cache\") pod \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") "
Sep 09 20:50:24.083804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.083797    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tmp-dir\") pod \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") "
Sep 09 20:50:24.084181 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.083823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcz64\" (UniqueName: \"kubernetes.io/projected/8139bf0b-c457-4fc3-9c1e-a37e58f09924-kube-api-access-vcz64\") pod \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") "
Sep 09 20:50:24.084181 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.083917    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-dshm\") pod \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\" (UID: \"8139bf0b-c457-4fc3-9c1e-a37e58f09924\") "
Sep 09 20:50:24.084181 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.083989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-model-cache" (OuterVolumeSpecName: "model-cache") pod "8139bf0b-c457-4fc3-9c1e-a37e58f09924" (UID: "8139bf0b-c457-4fc3-9c1e-a37e58f09924"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.084181 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.084018    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-home" (OuterVolumeSpecName: "home") pod "8139bf0b-c457-4fc3-9c1e-a37e58f09924" (UID: "8139bf0b-c457-4fc3-9c1e-a37e58f09924"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.084181 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.084056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8139bf0b-c457-4fc3-9c1e-a37e58f09924" (UID: "8139bf0b-c457-4fc3-9c1e-a37e58f09924"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.084402 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.084294    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:24.084402 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.084317    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:24.084402 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.084335    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:24.086160 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.086134    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8139bf0b-c457-4fc3-9c1e-a37e58f09924" (UID: "8139bf0b-c457-4fc3-9c1e-a37e58f09924"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:24.086282 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.086204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-dshm" (OuterVolumeSpecName: "dshm") pod "8139bf0b-c457-4fc3-9c1e-a37e58f09924" (UID: "8139bf0b-c457-4fc3-9c1e-a37e58f09924"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.086282 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.086214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8139bf0b-c457-4fc3-9c1e-a37e58f09924-kube-api-access-vcz64" (OuterVolumeSpecName: "kube-api-access-vcz64") pod "8139bf0b-c457-4fc3-9c1e-a37e58f09924" (UID: "8139bf0b-c457-4fc3-9c1e-a37e58f09924"). InnerVolumeSpecName "kube-api-access-vcz64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:24.185176 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.185145    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8139bf0b-c457-4fc3-9c1e-a37e58f09924-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:24.185176 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.185169    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8139bf0b-c457-4fc3-9c1e-a37e58f09924-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:24.185176 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.185179    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcz64\" (UniqueName: \"kubernetes.io/projected/8139bf0b-c457-4fc3-9c1e-a37e58f09924-kube-api-access-vcz64\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:24.602219 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.602172    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5" podUID="c11c6084-7cfe-4c79-b643-4604ac40ebd0" containerName="main" containerID="cri-o://8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19" gracePeriod=45
Sep 09 20:50:24.614858 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.614832    2566 generic.go:356] "Generic (PLEG): container finished" podID="8139bf0b-c457-4fc3-9c1e-a37e58f09924" containerID="983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200" exitCode=0
Sep 09 20:50:24.614971 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.614868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r" event={"ID":"8139bf0b-c457-4fc3-9c1e-a37e58f09924","Type":"ContainerDied","Data":"983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200"}
Sep 09 20:50:24.614971 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.614910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r" event={"ID":"8139bf0b-c457-4fc3-9c1e-a37e58f09924","Type":"ContainerDied","Data":"1328661cc0bb96bd4e51535521d4f068a6bce6b6c1b11ff69ca091b24953fb39"}
Sep 09 20:50:24.614971 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.614931    2566 scope.go:117] "RemoveContainer" containerID="983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200"
Sep 09 20:50:24.614971 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.614945    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"
Sep 09 20:50:24.663987 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.663954    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"]
Sep 09 20:50:24.670158 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.670133    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-4s26r"]
Sep 09 20:50:24.714660 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.714639    2566 scope.go:117] "RemoveContainer" containerID="983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200"
Sep 09 20:50:24.714996 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:50:24.714976    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200\": container with ID starting with 983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200 not found: ID does not exist" containerID="983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200"
Sep 09 20:50:24.715057 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.715011    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200"} err="failed to get container status \"983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200\": rpc error: code = NotFound desc = could not find container \"983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200\": container with ID starting with 983969f473fa665ce7e6ec2719babef2481dd7dba0d92ce78a243995a02c3200 not found: ID does not exist"
Sep 09 20:50:24.853907 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.853841    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:50:24.973247 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.973200    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q" podUID="91934607-2845-408b-9807-f60c80bff911" containerName="main" containerID="cri-o://b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc" gracePeriod=45
Sep 09 20:50:24.992615 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-dshm\") pod \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") "
Sep 09 20:50:24.992721 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-model-cache\") pod \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") "
Sep 09 20:50:24.992721 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992679    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tmp-dir\") pod \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") "
Sep 09 20:50:24.992721 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fkjm\" (UniqueName: \"kubernetes.io/projected/c11c6084-7cfe-4c79-b643-4604ac40ebd0-kube-api-access-8fkjm\") pod \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") "
Sep 09 20:50:24.992848 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992762    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tls-certs\") pod \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") "
Sep 09 20:50:24.992848 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-home\") pod \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\" (UID: \"c11c6084-7cfe-4c79-b643-4604ac40ebd0\") "
Sep 09 20:50:24.993004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c11c6084-7cfe-4c79-b643-4604ac40ebd0" (UID: "c11c6084-7cfe-4c79-b643-4604ac40ebd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.993004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.992970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c11c6084-7cfe-4c79-b643-4604ac40ebd0" (UID: "c11c6084-7cfe-4c79-b643-4604ac40ebd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.993203 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.993158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-home" (OuterVolumeSpecName: "home") pod "c11c6084-7cfe-4c79-b643-4604ac40ebd0" (UID: "c11c6084-7cfe-4c79-b643-4604ac40ebd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.994905 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.994872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-dshm" (OuterVolumeSpecName: "dshm") pod "c11c6084-7cfe-4c79-b643-4604ac40ebd0" (UID: "c11c6084-7cfe-4c79-b643-4604ac40ebd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:24.994980 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.994944    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11c6084-7cfe-4c79-b643-4604ac40ebd0-kube-api-access-8fkjm" (OuterVolumeSpecName: "kube-api-access-8fkjm") pod "c11c6084-7cfe-4c79-b643-4604ac40ebd0" (UID: "c11c6084-7cfe-4c79-b643-4604ac40ebd0"). InnerVolumeSpecName "kube-api-access-8fkjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:24.995321 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:24.995299    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c11c6084-7cfe-4c79-b643-4604ac40ebd0" (UID: "c11c6084-7cfe-4c79-b643-4604ac40ebd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:25.094177 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.094148    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.094177 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.094176    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.094328 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.094186    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fkjm\" (UniqueName: \"kubernetes.io/projected/c11c6084-7cfe-4c79-b643-4604ac40ebd0-kube-api-access-8fkjm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.094328 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.094196    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c11c6084-7cfe-4c79-b643-4604ac40ebd0-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.094328 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.094204    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.094328 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.094210    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c11c6084-7cfe-4c79-b643-4604ac40ebd0-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.125338 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.125298    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q" podUID="91934607-2845-408b-9807-f60c80bff911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 09 20:50:25.213480 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.213454    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:50:25.295446 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-tmp-dir\") pod \"91934607-2845-408b-9807-f60c80bff911\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") "
Sep 09 20:50:25.295446 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295446    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-dshm\") pod \"91934607-2845-408b-9807-f60c80bff911\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") "
Sep 09 20:50:25.295675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295462    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-home\") pod \"91934607-2845-408b-9807-f60c80bff911\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") "
Sep 09 20:50:25.295675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-model-cache\") pod \"91934607-2845-408b-9807-f60c80bff911\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") "
Sep 09 20:50:25.295675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d5zb\" (UniqueName: \"kubernetes.io/projected/91934607-2845-408b-9807-f60c80bff911-kube-api-access-5d5zb\") pod \"91934607-2845-408b-9807-f60c80bff911\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") "
Sep 09 20:50:25.295675 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295612    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91934607-2845-408b-9807-f60c80bff911-tls-certs\") pod \"91934607-2845-408b-9807-f60c80bff911\" (UID: \"91934607-2845-408b-9807-f60c80bff911\") "
Sep 09 20:50:25.295873 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295739    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91934607-2845-408b-9807-f60c80bff911" (UID: "91934607-2845-408b-9807-f60c80bff911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:25.295873 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295754    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-home" (OuterVolumeSpecName: "home") pod "91934607-2845-408b-9807-f60c80bff911" (UID: "91934607-2845-408b-9807-f60c80bff911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:25.295873 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295809    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-model-cache" (OuterVolumeSpecName: "model-cache") pod "91934607-2845-408b-9807-f60c80bff911" (UID: "91934607-2845-408b-9807-f60c80bff911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:25.296025 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295879    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.296025 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295892    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.296025 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.295900    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.297746 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.297722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-dshm" (OuterVolumeSpecName: "dshm") pod "91934607-2845-408b-9807-f60c80bff911" (UID: "91934607-2845-408b-9807-f60c80bff911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:25.297864 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.297823    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91934607-2845-408b-9807-f60c80bff911-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91934607-2845-408b-9807-f60c80bff911" (UID: "91934607-2845-408b-9807-f60c80bff911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:25.297864 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.297824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91934607-2845-408b-9807-f60c80bff911-kube-api-access-5d5zb" (OuterVolumeSpecName: "kube-api-access-5d5zb") pod "91934607-2845-408b-9807-f60c80bff911" (UID: "91934607-2845-408b-9807-f60c80bff911"). InnerVolumeSpecName "kube-api-access-5d5zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:25.396400 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.396311    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5zb\" (UniqueName: \"kubernetes.io/projected/91934607-2845-408b-9807-f60c80bff911-kube-api-access-5d5zb\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.396400 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.396344    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91934607-2845-408b-9807-f60c80bff911-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.396400 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.396354    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91934607-2845-408b-9807-f60c80bff911-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:50:25.493064 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.493020    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8139bf0b-c457-4fc3-9c1e-a37e58f09924" path="/var/lib/kubelet/pods/8139bf0b-c457-4fc3-9c1e-a37e58f09924/volumes"
Sep 09 20:50:25.620542 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.620505    2566 generic.go:356] "Generic (PLEG): container finished" podID="91934607-2845-408b-9807-f60c80bff911" containerID="b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc" exitCode=0
Sep 09 20:50:25.620746 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.620548    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"
Sep 09 20:50:25.620746 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.620589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q" event={"ID":"91934607-2845-408b-9807-f60c80bff911","Type":"ContainerDied","Data":"b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc"}
Sep 09 20:50:25.620746 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.620649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q" event={"ID":"91934607-2845-408b-9807-f60c80bff911","Type":"ContainerDied","Data":"96fc903ca9a8fcd675e8e08bc46cfb36123c43ac4a2033f1b3d3700094a61134"}
Sep 09 20:50:25.620746 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.620677    2566 scope.go:117] "RemoveContainer" containerID="b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc"
Sep 09 20:50:25.623113 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.623069    2566 generic.go:356] "Generic (PLEG): container finished" podID="c11c6084-7cfe-4c79-b643-4604ac40ebd0" containerID="8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19" exitCode=0
Sep 09 20:50:25.623236 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.623125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5" event={"ID":"c11c6084-7cfe-4c79-b643-4604ac40ebd0","Type":"ContainerDied","Data":"8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19"}
Sep 09 20:50:25.623236 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.623147    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"
Sep 09 20:50:25.623236 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.623171    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5" event={"ID":"c11c6084-7cfe-4c79-b643-4604ac40ebd0","Type":"ContainerDied","Data":"52bbc424be6c9709aee37bc7800207df839f2545ad703b143c842159ae22f724"}
Sep 09 20:50:25.629481 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.629462    2566 scope.go:117] "RemoveContainer" containerID="b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc"
Sep 09 20:50:25.629756 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:50:25.629737    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc\": container with ID starting with b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc not found: ID does not exist" containerID="b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc"
Sep 09 20:50:25.629815 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.629764    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc"} err="failed to get container status \"b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc\": rpc error: code = NotFound desc = could not find container \"b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc\": container with ID starting with b2ddca38bdcdb0a20db410d6072bdf5263b8103da26d60a0e36f1a310e4a53cc not found: ID does not exist"
Sep 09 20:50:25.629815 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.629781    2566 scope.go:117] "RemoveContainer" containerID="8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19"
Sep 09 20:50:25.637197 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.637173    2566 scope.go:117] "RemoveContainer" containerID="8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19"
Sep 09 20:50:25.637452 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:50:25.637434    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19\": container with ID starting with 8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19 not found: ID does not exist" containerID="8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19"
Sep 09 20:50:25.637515 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.637461    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19"} err="failed to get container status \"8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19\": rpc error: code = NotFound desc = could not find container \"8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19\": container with ID starting with 8dbcf13d1b3aaa0cd4c6ae8d26a3a83472424b92de18be47d1f232e22ac5aa19 not found: ID does not exist"
Sep 09 20:50:25.640548 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.640526    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"]
Sep 09 20:50:25.644906 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.644888    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-zgmt5"]
Sep 09 20:50:25.656154 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.656101    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"]
Sep 09 20:50:25.660755 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:25.660736    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-f8j8q"]
Sep 09 20:50:27.491647 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:27.491603    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91934607-2845-408b-9807-f60c80bff911" path="/var/lib/kubelet/pods/91934607-2845-408b-9807-f60c80bff911/volumes"
Sep 09 20:50:27.492178 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:27.492158    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c11c6084-7cfe-4c79-b643-4604ac40ebd0" path="/var/lib/kubelet/pods/c11c6084-7cfe-4c79-b643-4604ac40ebd0/volumes"
Sep 09 20:50:41.353104 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353037    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"]
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353425    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353447    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353459    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11c6084-7cfe-4c79-b643-4604ac40ebd0" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353465    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11c6084-7cfe-4c79-b643-4604ac40ebd0" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353473    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerName="storage-initializer"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353478    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerName="storage-initializer"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353489    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91934607-2845-408b-9807-f60c80bff911" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353494    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91934607-2845-408b-9807-f60c80bff911" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353504    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8139bf0b-c457-4fc3-9c1e-a37e58f09924" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353508    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8139bf0b-c457-4fc3-9c1e-a37e58f09924" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353559    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91934607-2845-408b-9807-f60c80bff911" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353566    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="34185ff8-e6a0-4d8e-94d9-def3d143fca2" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353572    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11c6084-7cfe-4c79-b643-4604ac40ebd0" containerName="main"
Sep 09 20:50:41.353572 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.353579    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8139bf0b-c457-4fc3-9c1e-a37e58f09924" containerName="main"
Sep 09 20:50:41.357520 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.357501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.360056 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.360034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 09 20:50:41.361234 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.361208    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 09 20:50:41.361342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.361214    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-vnfmt\""
Sep 09 20:50:41.361342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.361218    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 09 20:50:41.367317 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.367290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"]
Sep 09 20:50:41.431669 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.431641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-model-cache\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.431815 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.431693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4r8p\" (UniqueName: \"kubernetes.io/projected/079b1c44-2bd9-4016-8500-a3f16c7dcf73-kube-api-access-g4r8p\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.431815 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.431769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-home\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.431815 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.431803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-dshm\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.431922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.431845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tls-certs\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.431922 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.431866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.532503 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.532459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4r8p\" (UniqueName: \"kubernetes.io/projected/079b1c44-2bd9-4016-8500-a3f16c7dcf73-kube-api-access-g4r8p\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.532688 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.532514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-home\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.532688 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.532548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-dshm\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.532688 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.532595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tls-certs\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.532688 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.532619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.532688 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.532648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-model-cache\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.533190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.533021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-home\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.533190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.533035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.533190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.533126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-model-cache\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.534955 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.534934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-dshm\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.535129 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.535112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tls-certs\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.541193 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.541171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4r8p\" (UniqueName: \"kubernetes.io/projected/079b1c44-2bd9-4016-8500-a3f16c7dcf73-kube-api-access-g4r8p\") pod \"late-v1-kserve-56cbc8456c-rh9w5\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.669637 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.669601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:41.946318 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:41.946292    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"]
Sep 09 20:50:41.949074 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:50:41.949037    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079b1c44_2bd9_4016_8500_a3f16c7dcf73.slice/crio-b1a0567e2e077c17b6e93b671efd10f69a09b3c3d9afca81baca2b401f6287f1 WatchSource:0}: Error finding container b1a0567e2e077c17b6e93b671efd10f69a09b3c3d9afca81baca2b401f6287f1: Status 404 returned error can't find the container with id b1a0567e2e077c17b6e93b671efd10f69a09b3c3d9afca81baca2b401f6287f1
Sep 09 20:50:42.705417 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:42.705384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5" event={"ID":"079b1c44-2bd9-4016-8500-a3f16c7dcf73","Type":"ContainerStarted","Data":"150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf"}
Sep 09 20:50:42.705841 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:42.705441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5" event={"ID":"079b1c44-2bd9-4016-8500-a3f16c7dcf73","Type":"ContainerStarted","Data":"b1a0567e2e077c17b6e93b671efd10f69a09b3c3d9afca81baca2b401f6287f1"}
Sep 09 20:50:42.723208 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:42.723155    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5" podStartSLOduration=1.723138066 podStartE2EDuration="1.723138066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:42.721653308 +0000 UTC m=+2367.757124468" watchObservedRunningTime="2026-09-09 20:50:42.723138066 +0000 UTC m=+2367.758609224"
Sep 09 20:50:43.670689 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:43.670650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:51.669939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:51.669896    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:51.681735 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:51.681713    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:51.763271 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:51.763241    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:50:58.040382 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.040304    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"]
Sep 09 20:50:58.044074 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.044053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.046724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.046704    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 09 20:50:58.046991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.046978    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:50:58.047707 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.047692    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 09 20:50:58.047768 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.047726    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 09 20:50:58.047891 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.047877    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-k776l\""
Sep 09 20:50:58.047960 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.047938    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 09 20:50:58.054865 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.054840    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"]
Sep 09 20:50:58.079236 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.079378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.079378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.079378 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tls-certs\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.079500 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-home\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.079500 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-dshm\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.079500 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-model-cache\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.079500 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.079469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lk2\" (UniqueName: \"kubernetes.io/projected/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kube-api-access-v4lk2\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180510 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lk2\" (UniqueName: \"kubernetes.io/projected/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kube-api-access-v4lk2\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180682 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180734 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180769 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180806 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tls-certs\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180870 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-home\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180908 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180946 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-dshm\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.180982 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.180938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-model-cache\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.181064 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.181045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tmp-dir\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.181185 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.181167    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-home\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.181342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.181318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-model-cache\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.181498 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.181477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.183304 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.183277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-dshm\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.183423 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.183407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tls-certs\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.189927 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.189909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lk2\" (UniqueName: \"kubernetes.io/projected/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kube-api-access-v4lk2\") pod \"tokenizer-template-args-test-kserve-84c478d7c9-9cb6v\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.355233 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.355137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:50:58.632583 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.632558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"]
Sep 09 20:50:58.635408 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:50:58.635375    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe0059c_35ee_4d76_b1c6_ab95f17b4316.slice/crio-0a8c803cad06b71d8dee77c9c7771ab3036f60c7bb7078bb188d2df20cfaf76c WatchSource:0}: Error finding container 0a8c803cad06b71d8dee77c9c7771ab3036f60c7bb7078bb188d2df20cfaf76c: Status 404 returned error can't find the container with id 0a8c803cad06b71d8dee77c9c7771ab3036f60c7bb7078bb188d2df20cfaf76c
Sep 09 20:50:58.791223 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.791184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" event={"ID":"2fe0059c-35ee-4d76-b1c6-ab95f17b4316","Type":"ContainerStarted","Data":"acdd0b27b87fc7bb84b11f9fb16841f4be3f27072f33f36c82b1ba38093d81e8"}
Sep 09 20:50:58.791393 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:50:58.791245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" event={"ID":"2fe0059c-35ee-4d76-b1c6-ab95f17b4316","Type":"ContainerStarted","Data":"0a8c803cad06b71d8dee77c9c7771ab3036f60c7bb7078bb188d2df20cfaf76c"}
Sep 09 20:51:01.810333 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:01.810295    2566 generic.go:356] "Generic (PLEG): container finished" podID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerID="acdd0b27b87fc7bb84b11f9fb16841f4be3f27072f33f36c82b1ba38093d81e8" exitCode=0
Sep 09 20:51:01.810696 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:01.810354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" event={"ID":"2fe0059c-35ee-4d76-b1c6-ab95f17b4316","Type":"ContainerDied","Data":"acdd0b27b87fc7bb84b11f9fb16841f4be3f27072f33f36c82b1ba38093d81e8"}
Sep 09 20:51:02.817682 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:02.817645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" event={"ID":"2fe0059c-35ee-4d76-b1c6-ab95f17b4316","Type":"ContainerStarted","Data":"19ce7cc88026765b61cee9032cec77b9acfeb47458f73b158be3848b97ff0000"}
Sep 09 20:51:02.839342 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:02.839290    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" podStartSLOduration=2.612889182 podStartE2EDuration="4.83927355s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.226384368s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:02.837896745 +0000 UTC m=+2387.873367905" watchObservedRunningTime="2026-09-09 20:51:02.83927355 +0000 UTC m=+2387.874744708"
Sep 09 20:51:03.355531 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:03.355494    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:51:08.355413 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:08.355372    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:51:08.367629 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:08.367599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:51:08.857856 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:08.857819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:51:34.650005 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.649971    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"]
Sep 09 20:51:34.653653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.653634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.656455 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.656435    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 09 20:51:34.662841 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.662818    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"]
Sep 09 20:51:34.813352 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.813314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-tmp-dir\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.813558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.813366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-home\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.813558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.813463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vztnj\" (UniqueName: \"kubernetes.io/projected/1861f35a-c68c-444c-bdab-3826af880c85-kube-api-access-vztnj\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.813558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.813507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-dshm\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.813558 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.813530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-model-cache\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.813720 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.813566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1861f35a-c68c-444c-bdab-3826af880c85-tls-certs\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.914494 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-tmp-dir\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.914494 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-home\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.914783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vztnj\" (UniqueName: \"kubernetes.io/projected/1861f35a-c68c-444c-bdab-3826af880c85-kube-api-access-vztnj\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.914783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-dshm\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.914783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-model-cache\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.914783 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1861f35a-c68c-444c-bdab-3826af880c85-tls-certs\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.914977 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-tmp-dir\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.915036 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.914992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-home\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.915118 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.915064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-model-cache\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.917177 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.917144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-dshm\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.917387 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.917366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1861f35a-c68c-444c-bdab-3826af880c85-tls-certs\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.922665 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.922641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vztnj\" (UniqueName: \"kubernetes.io/projected/1861f35a-c68c-444c-bdab-3826af880c85-kube-api-access-vztnj\") pod \"late-v2-kserve-86766546f-pk75p\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:34.964640 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:34.964613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:35.235347 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:35.235318    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"]
Sep 09 20:51:35.237474 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:51:35.237432    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1861f35a_c68c_444c_bdab_3826af880c85.slice/crio-8d84b77c66dd4df60b8b344b5b66d98d6961cdf49120ac4aaba2798335991254 WatchSource:0}: Error finding container 8d84b77c66dd4df60b8b344b5b66d98d6961cdf49120ac4aaba2798335991254: Status 404 returned error can't find the container with id 8d84b77c66dd4df60b8b344b5b66d98d6961cdf49120ac4aaba2798335991254
Sep 09 20:51:35.968858 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:35.968821    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p" event={"ID":"1861f35a-c68c-444c-bdab-3826af880c85","Type":"ContainerStarted","Data":"770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564"}
Sep 09 20:51:35.969240 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:35.968876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p" event={"ID":"1861f35a-c68c-444c-bdab-3826af880c85","Type":"ContainerStarted","Data":"8d84b77c66dd4df60b8b344b5b66d98d6961cdf49120ac4aaba2798335991254"}
Sep 09 20:51:35.986142 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:35.986052    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p" podStartSLOduration=1.986037941 podStartE2EDuration="1.986037941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:35.984922343 +0000 UTC m=+2421.020393501" watchObservedRunningTime="2026-09-09 20:51:35.986037941 +0000 UTC m=+2421.021509099"
Sep 09 20:51:36.965323 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:36.965237    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:44.965332 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:44.965294    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:44.977368 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:44.977342    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:45.022551 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:45.022521    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:51:53.767976 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:53.767936    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"]
Sep 09 20:51:53.777848 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:51:53.777801    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:51:53.777848 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:51:53.777848 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:51:53.777848 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="19ce7cc88026765b61cee9032cec77b9acfeb47458f73b158be3848b97ff0000" cmd=["/bin/sleep","15"]
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:53.777865    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" message="" err=<
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:51:53.777882    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="main" containerID="cri-o://19ce7cc88026765b61cee9032cec77b9acfeb47458f73b158be3848b97ff0000"
Sep 09 20:51:53.778204 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:53.777955    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="main" containerID="cri-o://19ce7cc88026765b61cee9032cec77b9acfeb47458f73b158be3848b97ff0000" gracePeriod=60
Sep 09 20:51:53.847887 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:53.847846    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 09 20:51:54.054754 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.054719    2566 generic.go:356] "Generic (PLEG): container finished" podID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerID="19ce7cc88026765b61cee9032cec77b9acfeb47458f73b158be3848b97ff0000" exitCode=0
Sep 09 20:51:54.054888 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.054795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" event={"ID":"2fe0059c-35ee-4d76-b1c6-ab95f17b4316","Type":"ContainerDied","Data":"19ce7cc88026765b61cee9032cec77b9acfeb47458f73b158be3848b97ff0000"}
Sep 09 20:51:54.132724 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.132701    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:51:54.194005 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.193978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tmp-dir\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194199 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-cabundle-cert\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194199 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194044    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-home\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194199 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194060    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4lk2\" (UniqueName: \"kubernetes.io/projected/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kube-api-access-v4lk2\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194202    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tls-certs\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194237    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-dshm\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194274    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-model-cache\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kserve-provision-location\") pod \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\" (UID: \"2fe0059c-35ee-4d76-b1c6-ab95f17b4316\") "
Sep 09 20:51:54.194386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.194386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194330    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-home" (OuterVolumeSpecName: "home") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.194680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194408    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:54.194680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194571    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.194680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194589    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.194680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194608    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.194680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.194622    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.196604 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.196577    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:51:54.196712 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.196617    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kube-api-access-v4lk2" (OuterVolumeSpecName: "kube-api-access-v4lk2") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "kube-api-access-v4lk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:54.196758 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.196725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-dshm" (OuterVolumeSpecName: "dshm") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.215132 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.215107    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2fe0059c-35ee-4d76-b1c6-ab95f17b4316" (UID: "2fe0059c-35ee-4d76-b1c6-ab95f17b4316"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:54.295804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.295723    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lk2\" (UniqueName: \"kubernetes.io/projected/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kube-api-access-v4lk2\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.295804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.295750    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.295804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.295762    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.295804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.295774    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:54.295804 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:54.295785    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe0059c-35ee-4d76-b1c6-ab95f17b4316-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:51:55.061319 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:55.061289    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"
Sep 09 20:51:55.061319 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:55.061305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v" event={"ID":"2fe0059c-35ee-4d76-b1c6-ab95f17b4316","Type":"ContainerDied","Data":"0a8c803cad06b71d8dee77c9c7771ab3036f60c7bb7078bb188d2df20cfaf76c"}
Sep 09 20:51:55.061787 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:55.061393    2566 scope.go:117] "RemoveContainer" containerID="19ce7cc88026765b61cee9032cec77b9acfeb47458f73b158be3848b97ff0000"
Sep 09 20:51:55.071054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:55.071037    2566 scope.go:117] "RemoveContainer" containerID="acdd0b27b87fc7bb84b11f9fb16841f4be3f27072f33f36c82b1ba38093d81e8"
Sep 09 20:51:55.084160 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:55.084135    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"]
Sep 09 20:51:55.088789 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:55.088764    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-84c478d7c9-9cb6v"]
Sep 09 20:51:55.494192 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:51:55.494159    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" path="/var/lib/kubelet/pods/2fe0059c-35ee-4d76-b1c6-ab95f17b4316/volumes"
Sep 09 20:52:20.310350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:20.310307    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"]
Sep 09 20:52:21.137466 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:21.137422    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"]
Sep 09 20:52:35.331773 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.331720    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5" podUID="079b1c44-2bd9-4016-8500-a3f16c7dcf73" containerName="main" containerID="cri-o://150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf" gracePeriod=45
Sep 09 20:52:35.590105 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.590025    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:52:35.653634 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653606    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-home\") pod \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") "
Sep 09 20:52:35.653792 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tls-certs\") pod \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") "
Sep 09 20:52:35.653792 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tmp-dir\") pod \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") "
Sep 09 20:52:35.653792 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-dshm\") pod \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") "
Sep 09 20:52:35.653792 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653783    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-model-cache\") pod \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") "
Sep 09 20:52:35.653998 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4r8p\" (UniqueName: \"kubernetes.io/projected/079b1c44-2bd9-4016-8500-a3f16c7dcf73-kube-api-access-g4r8p\") pod \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\" (UID: \"079b1c44-2bd9-4016-8500-a3f16c7dcf73\") "
Sep 09 20:52:35.653998 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653920    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-home" (OuterVolumeSpecName: "home") pod "079b1c44-2bd9-4016-8500-a3f16c7dcf73" (UID: "079b1c44-2bd9-4016-8500-a3f16c7dcf73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:35.653998 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.653964    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "079b1c44-2bd9-4016-8500-a3f16c7dcf73" (UID: "079b1c44-2bd9-4016-8500-a3f16c7dcf73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:35.654190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.654133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-model-cache" (OuterVolumeSpecName: "model-cache") pod "079b1c44-2bd9-4016-8500-a3f16c7dcf73" (UID: "079b1c44-2bd9-4016-8500-a3f16c7dcf73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:35.654190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.654146    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:35.654190 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.654166    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:35.656124 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.656104    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079b1c44-2bd9-4016-8500-a3f16c7dcf73-kube-api-access-g4r8p" (OuterVolumeSpecName: "kube-api-access-g4r8p") pod "079b1c44-2bd9-4016-8500-a3f16c7dcf73" (UID: "079b1c44-2bd9-4016-8500-a3f16c7dcf73"). InnerVolumeSpecName "kube-api-access-g4r8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:35.656212 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.656147    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-dshm" (OuterVolumeSpecName: "dshm") pod "079b1c44-2bd9-4016-8500-a3f16c7dcf73" (UID: "079b1c44-2bd9-4016-8500-a3f16c7dcf73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:35.656212 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.656197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "079b1c44-2bd9-4016-8500-a3f16c7dcf73" (UID: "079b1c44-2bd9-4016-8500-a3f16c7dcf73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:52:35.755019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.754983    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/079b1c44-2bd9-4016-8500-a3f16c7dcf73-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:35.755019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.755011    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:35.755019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.755023    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/079b1c44-2bd9-4016-8500-a3f16c7dcf73-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:35.755249 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:35.755037    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4r8p\" (UniqueName: \"kubernetes.io/projected/079b1c44-2bd9-4016-8500-a3f16c7dcf73-kube-api-access-g4r8p\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:36.158248 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.158204    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p" podUID="1861f35a-c68c-444c-bdab-3826af880c85" containerName="main" containerID="cri-o://770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564" gracePeriod=45
Sep 09 20:52:36.259395 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.259356    2566 generic.go:356] "Generic (PLEG): container finished" podID="079b1c44-2bd9-4016-8500-a3f16c7dcf73" containerID="150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf" exitCode=0
Sep 09 20:52:36.259570 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.259402    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"
Sep 09 20:52:36.259570 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.259418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5" event={"ID":"079b1c44-2bd9-4016-8500-a3f16c7dcf73","Type":"ContainerDied","Data":"150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf"}
Sep 09 20:52:36.259570 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.259467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5" event={"ID":"079b1c44-2bd9-4016-8500-a3f16c7dcf73","Type":"ContainerDied","Data":"b1a0567e2e077c17b6e93b671efd10f69a09b3c3d9afca81baca2b401f6287f1"}
Sep 09 20:52:36.259570 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.259489    2566 scope.go:117] "RemoveContainer" containerID="150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf"
Sep 09 20:52:36.269808 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.269785    2566 scope.go:117] "RemoveContainer" containerID="150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf"
Sep 09 20:52:36.270124 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:52:36.270101    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf\": container with ID starting with 150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf not found: ID does not exist" containerID="150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf"
Sep 09 20:52:36.270225 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.270132    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf"} err="failed to get container status \"150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf\": rpc error: code = NotFound desc = could not find container \"150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf\": container with ID starting with 150c494ee08ab5fb7314da8d8fe78c1daada1147b7f8893cfe11bb7ddccad8bf not found: ID does not exist"
Sep 09 20:52:36.282473 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.282443    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"]
Sep 09 20:52:36.286593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.286566    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rh9w5"]
Sep 09 20:52:36.404796 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.404771    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:52:36.460267 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1861f35a-c68c-444c-bdab-3826af880c85-tls-certs\") pod \"1861f35a-c68c-444c-bdab-3826af880c85\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") "
Sep 09 20:52:36.460267 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-tmp-dir\") pod \"1861f35a-c68c-444c-bdab-3826af880c85\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") "
Sep 09 20:52:36.460267 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-dshm\") pod \"1861f35a-c68c-444c-bdab-3826af880c85\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") "
Sep 09 20:52:36.460560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-home\") pod \"1861f35a-c68c-444c-bdab-3826af880c85\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") "
Sep 09 20:52:36.460560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460297    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-model-cache\") pod \"1861f35a-c68c-444c-bdab-3826af880c85\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") "
Sep 09 20:52:36.460560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vztnj\" (UniqueName: \"kubernetes.io/projected/1861f35a-c68c-444c-bdab-3826af880c85-kube-api-access-vztnj\") pod \"1861f35a-c68c-444c-bdab-3826af880c85\" (UID: \"1861f35a-c68c-444c-bdab-3826af880c85\") "
Sep 09 20:52:36.460560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1861f35a-c68c-444c-bdab-3826af880c85" (UID: "1861f35a-c68c-444c-bdab-3826af880c85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:36.460560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-home" (OuterVolumeSpecName: "home") pod "1861f35a-c68c-444c-bdab-3826af880c85" (UID: "1861f35a-c68c-444c-bdab-3826af880c85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:36.460761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460579    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-model-cache" (OuterVolumeSpecName: "model-cache") pod "1861f35a-c68c-444c-bdab-3826af880c85" (UID: "1861f35a-c68c-444c-bdab-3826af880c85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:36.460761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460670    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:36.460761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460683    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:36.460761 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.460692    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:36.462623 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.462598    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1861f35a-c68c-444c-bdab-3826af880c85-kube-api-access-vztnj" (OuterVolumeSpecName: "kube-api-access-vztnj") pod "1861f35a-c68c-444c-bdab-3826af880c85" (UID: "1861f35a-c68c-444c-bdab-3826af880c85"). InnerVolumeSpecName "kube-api-access-vztnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:36.462726 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.462622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1861f35a-c68c-444c-bdab-3826af880c85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1861f35a-c68c-444c-bdab-3826af880c85" (UID: "1861f35a-c68c-444c-bdab-3826af880c85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:52:36.462726 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.462702    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-dshm" (OuterVolumeSpecName: "dshm") pod "1861f35a-c68c-444c-bdab-3826af880c85" (UID: "1861f35a-c68c-444c-bdab-3826af880c85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:36.561456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.561410    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1861f35a-c68c-444c-bdab-3826af880c85-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:36.561456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.561451    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1861f35a-c68c-444c-bdab-3826af880c85-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:36.561456 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:36.561464    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vztnj\" (UniqueName: \"kubernetes.io/projected/1861f35a-c68c-444c-bdab-3826af880c85-kube-api-access-vztnj\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:52:37.264992 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.264955    2566 generic.go:356] "Generic (PLEG): container finished" podID="1861f35a-c68c-444c-bdab-3826af880c85" containerID="770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564" exitCode=0
Sep 09 20:52:37.265213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.265003    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"
Sep 09 20:52:37.265213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.265046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p" event={"ID":"1861f35a-c68c-444c-bdab-3826af880c85","Type":"ContainerDied","Data":"770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564"}
Sep 09 20:52:37.265213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.265123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p" event={"ID":"1861f35a-c68c-444c-bdab-3826af880c85","Type":"ContainerDied","Data":"8d84b77c66dd4df60b8b344b5b66d98d6961cdf49120ac4aaba2798335991254"}
Sep 09 20:52:37.265213 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.265149    2566 scope.go:117] "RemoveContainer" containerID="770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564"
Sep 09 20:52:37.274234 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.274216    2566 scope.go:117] "RemoveContainer" containerID="770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564"
Sep 09 20:52:37.274552 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:52:37.274534    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564\": container with ID starting with 770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564 not found: ID does not exist" containerID="770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564"
Sep 09 20:52:37.274610 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.274564    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564"} err="failed to get container status \"770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564\": rpc error: code = NotFound desc = could not find container \"770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564\": container with ID starting with 770ceebab69b9eb1c200ad1d87f8887aefff68666450fbe7a82c93109d447564 not found: ID does not exist"
Sep 09 20:52:37.288302 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.288275    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"]
Sep 09 20:52:37.293232 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.293206    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-pk75p"]
Sep 09 20:52:37.491077 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.491038    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079b1c44-2bd9-4016-8500-a3f16c7dcf73" path="/var/lib/kubelet/pods/079b1c44-2bd9-4016-8500-a3f16c7dcf73/volumes"
Sep 09 20:52:37.491654 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:37.491633    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1861f35a-c68c-444c-bdab-3826af880c85" path="/var/lib/kubelet/pods/1861f35a-c68c-444c-bdab-3826af880c85/volumes"
Sep 09 20:52:38.153282 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.153246    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"]
Sep 09 20:52:38.154340 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.154310    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="storage-initializer"
Sep 09 20:52:38.154522 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.154508    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="storage-initializer"
Sep 09 20:52:38.154672 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.154660    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1861f35a-c68c-444c-bdab-3826af880c85" containerName="main"
Sep 09 20:52:38.154752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.154742    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1861f35a-c68c-444c-bdab-3826af880c85" containerName="main"
Sep 09 20:52:38.154852 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.154842    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079b1c44-2bd9-4016-8500-a3f16c7dcf73" containerName="main"
Sep 09 20:52:38.154920 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.154911    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="079b1c44-2bd9-4016-8500-a3f16c7dcf73" containerName="main"
Sep 09 20:52:38.155015 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.155006    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="main"
Sep 09 20:52:38.155107 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.155078    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="main"
Sep 09 20:52:38.155401 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.155386    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe0059c-35ee-4d76-b1c6-ab95f17b4316" containerName="main"
Sep 09 20:52:38.155514 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.155504    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1861f35a-c68c-444c-bdab-3826af880c85" containerName="main"
Sep 09 20:52:38.155590 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.155581    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="079b1c44-2bd9-4016-8500-a3f16c7dcf73" containerName="main"
Sep 09 20:52:38.161151 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.161053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.164275 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.164253    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 09 20:52:38.165370 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.165334    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-gf9tw\""
Sep 09 20:52:38.165501 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.165338    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 09 20:52:38.165501 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.165391    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 09 20:52:38.170281 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.170253    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"]
Sep 09 20:52:38.275954 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.275923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-home\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.275954 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.275959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-dshm\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.276198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.275980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e2a14-bb32-4952-9a09-124e2d573025-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.276198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.276112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.276198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.276137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.276198 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.276172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4mm\" (UniqueName: \"kubernetes.io/projected/535e2a14-bb32-4952-9a09-124e2d573025-kube-api-access-qx4mm\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-dshm\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377391 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e2a14-bb32-4952-9a09-124e2d573025-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4mm\" (UniqueName: \"kubernetes.io/projected/535e2a14-bb32-4952-9a09-124e2d573025-kube-api-access-qx4mm\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377653 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-home\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377890 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377955 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.377994 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.377965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-home\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.379899 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.379880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-dshm\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.380076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.380059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e2a14-bb32-4952-9a09-124e2d573025-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.387027 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.386999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4mm\" (UniqueName: \"kubernetes.io/projected/535e2a14-bb32-4952-9a09-124e2d573025-kube-api-access-qx4mm\") pod \"delw-v1-kserve-6cf8cbb847-tlccm\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.474192 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.474111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:38.760521 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:38.760423    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"]
Sep 09 20:52:38.763809 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:52:38.763775    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535e2a14_bb32_4952_9a09_124e2d573025.slice/crio-b0d05d62a550a02d1023a2e19d2694f30a283a759a1aad4f3b74968004f63b13 WatchSource:0}: Error finding container b0d05d62a550a02d1023a2e19d2694f30a283a759a1aad4f3b74968004f63b13: Status 404 returned error can't find the container with id b0d05d62a550a02d1023a2e19d2694f30a283a759a1aad4f3b74968004f63b13
Sep 09 20:52:39.278777 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:39.278739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm" event={"ID":"535e2a14-bb32-4952-9a09-124e2d573025","Type":"ContainerStarted","Data":"a4094d2ae03f116b0b05c17f094a30b224af6d452168ac85a1b4597c92cbc1a3"}
Sep 09 20:52:39.279000 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:39.278791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm" event={"ID":"535e2a14-bb32-4952-9a09-124e2d573025","Type":"ContainerStarted","Data":"b0d05d62a550a02d1023a2e19d2694f30a283a759a1aad4f3b74968004f63b13"}
Sep 09 20:52:39.296538 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:39.296469    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm" podStartSLOduration=1.296449377 podStartE2EDuration="1.296449377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:39.295956938 +0000 UTC m=+2484.331428099" watchObservedRunningTime="2026-09-09 20:52:39.296449377 +0000 UTC m=+2484.331920537"
Sep 09 20:52:39.474639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:39.474595    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:48.475025 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:48.474985    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:48.488072 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:48.488037    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:52:49.336028 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:52:49.335999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:53:06.157020 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.156982    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"]
Sep 09 20:53:06.160349 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.160327    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.164440 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.164402    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-dv7j8\""
Sep 09 20:53:06.164440 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.164406    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 09 20:53:06.164639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.164440    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 09 20:53:06.164639 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.164460    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 09 20:53:06.172350 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.172324    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"]
Sep 09 20:53:06.327255 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.327213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb37bbad-ae79-41c6-ada9-81e2becc083e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.327458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.327283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.327458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.327321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.327458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.327355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmm6\" (UniqueName: \"kubernetes.io/projected/bb37bbad-ae79-41c6-ada9-81e2becc083e-kube-api-access-klmm6\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.327458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.327377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.327458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.327404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428390 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428390 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmm6\" (UniqueName: \"kubernetes.io/projected/bb37bbad-ae79-41c6-ada9-81e2becc083e-kube-api-access-klmm6\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb37bbad-ae79-41c6-ada9-81e2becc083e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428627 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428855 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428855 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.428969 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.428920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.430814 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.430793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.431066 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.431048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb37bbad-ae79-41c6-ada9-81e2becc083e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.440839 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.440815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmm6\" (UniqueName: \"kubernetes.io/projected/bb37bbad-ae79-41c6-ada9-81e2becc083e-kube-api-access-klmm6\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.471977 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.471940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:06.746117 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:06.746043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"]
Sep 09 20:53:06.748420 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:53:06.748384    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb37bbad_ae79_41c6_ada9_81e2becc083e.slice/crio-6c0faafc3b5a0d7c0f4816a614a6be079806fb93648d809fb70fe10545c810ec WatchSource:0}: Error finding container 6c0faafc3b5a0d7c0f4816a614a6be079806fb93648d809fb70fe10545c810ec: Status 404 returned error can't find the container with id 6c0faafc3b5a0d7c0f4816a614a6be079806fb93648d809fb70fe10545c810ec
Sep 09 20:53:07.417332 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:07.417289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" event={"ID":"bb37bbad-ae79-41c6-ada9-81e2becc083e","Type":"ContainerStarted","Data":"8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e"}
Sep 09 20:53:07.417740 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:07.417361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" event={"ID":"bb37bbad-ae79-41c6-ada9-81e2becc083e","Type":"ContainerStarted","Data":"6c0faafc3b5a0d7c0f4816a614a6be079806fb93648d809fb70fe10545c810ec"}
Sep 09 20:53:07.437866 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:07.437801    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" podStartSLOduration=1.437779028 podStartE2EDuration="1.437779028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:07.436059828 +0000 UTC m=+2512.471531020" watchObservedRunningTime="2026-09-09 20:53:07.437779028 +0000 UTC m=+2512.473250189"
Sep 09 20:53:07.472365 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:07.472326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:16.472429 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:16.472389    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:16.485426 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:16.485391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:17.474232 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:17.474200    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:39.730298 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:39.730256    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"]
Sep 09 20:53:39.740709 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:53:39.740661    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:53:39.740709 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:53:39.740709 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:53:39.740709 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e" cmd=["/bin/sleep","15"]
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:39.740723    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" message="" err=<
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:53:39.740735    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" podUID="bb37bbad-ae79-41c6-ada9-81e2becc083e" containerName="main" containerID="cri-o://8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e"
Sep 09 20:53:39.740991 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:39.740784    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" podUID="bb37bbad-ae79-41c6-ada9-81e2becc083e" containerName="main" containerID="cri-o://8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e" gracePeriod=60
Sep 09 20:53:40.107519 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.107486    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:40.236186 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-model-cache\") pod \"bb37bbad-ae79-41c6-ada9-81e2becc083e\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") "
Sep 09 20:53:40.236186 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb37bbad-ae79-41c6-ada9-81e2becc083e-tls-certs\") pod \"bb37bbad-ae79-41c6-ada9-81e2becc083e\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") "
Sep 09 20:53:40.236458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-tmp-dir\") pod \"bb37bbad-ae79-41c6-ada9-81e2becc083e\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") "
Sep 09 20:53:40.236458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236265    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-dshm\") pod \"bb37bbad-ae79-41c6-ada9-81e2becc083e\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") "
Sep 09 20:53:40.236458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236296    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-home\") pod \"bb37bbad-ae79-41c6-ada9-81e2becc083e\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") "
Sep 09 20:53:40.236458 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236356    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klmm6\" (UniqueName: \"kubernetes.io/projected/bb37bbad-ae79-41c6-ada9-81e2becc083e-kube-api-access-klmm6\") pod \"bb37bbad-ae79-41c6-ada9-81e2becc083e\" (UID: \"bb37bbad-ae79-41c6-ada9-81e2becc083e\") "
Sep 09 20:53:40.236668 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb37bbad-ae79-41c6-ada9-81e2becc083e" (UID: "bb37bbad-ae79-41c6-ada9-81e2becc083e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:40.236668 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-home" (OuterVolumeSpecName: "home") pod "bb37bbad-ae79-41c6-ada9-81e2becc083e" (UID: "bb37bbad-ae79-41c6-ada9-81e2becc083e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:40.236855 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.236827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb37bbad-ae79-41c6-ada9-81e2becc083e" (UID: "bb37bbad-ae79-41c6-ada9-81e2becc083e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:40.238892 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.238857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb37bbad-ae79-41c6-ada9-81e2becc083e-kube-api-access-klmm6" (OuterVolumeSpecName: "kube-api-access-klmm6") pod "bb37bbad-ae79-41c6-ada9-81e2becc083e" (UID: "bb37bbad-ae79-41c6-ada9-81e2becc083e"). InnerVolumeSpecName "kube-api-access-klmm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:40.239008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.238913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb37bbad-ae79-41c6-ada9-81e2becc083e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb37bbad-ae79-41c6-ada9-81e2becc083e" (UID: "bb37bbad-ae79-41c6-ada9-81e2becc083e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:40.239008 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.238983    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-dshm" (OuterVolumeSpecName: "dshm") pod "bb37bbad-ae79-41c6-ada9-81e2becc083e" (UID: "bb37bbad-ae79-41c6-ada9-81e2becc083e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:40.337830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.337739    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:53:40.337830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.337768    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:53:40.337830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.337778    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klmm6\" (UniqueName: \"kubernetes.io/projected/bb37bbad-ae79-41c6-ada9-81e2becc083e-kube-api-access-klmm6\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:53:40.337830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.337788    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:53:40.337830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.337798    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb37bbad-ae79-41c6-ada9-81e2becc083e-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:53:40.337830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.337806    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb37bbad-ae79-41c6-ada9-81e2becc083e-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:53:40.574874 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.574835    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb37bbad-ae79-41c6-ada9-81e2becc083e" containerID="8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e" exitCode=0
Sep 09 20:53:40.574874 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.574877    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"
Sep 09 20:53:40.575104 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.574899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" event={"ID":"bb37bbad-ae79-41c6-ada9-81e2becc083e","Type":"ContainerDied","Data":"8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e"}
Sep 09 20:53:40.575104 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.574948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc" event={"ID":"bb37bbad-ae79-41c6-ada9-81e2becc083e","Type":"ContainerDied","Data":"6c0faafc3b5a0d7c0f4816a614a6be079806fb93648d809fb70fe10545c810ec"}
Sep 09 20:53:40.575104 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.574973    2566 scope.go:117] "RemoveContainer" containerID="8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e"
Sep 09 20:53:40.584493 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.584474    2566 scope.go:117] "RemoveContainer" containerID="8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e"
Sep 09 20:53:40.584793 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:53:40.584773    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e\": container with ID starting with 8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e not found: ID does not exist" containerID="8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e"
Sep 09 20:53:40.584839 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.584803    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e"} err="failed to get container status \"8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e\": rpc error: code = NotFound desc = could not find container \"8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e\": container with ID starting with 8b4b71ed7825219952756bb8287ac2a004f4c873cee55d600e58300bde63dd0e not found: ID does not exist"
Sep 09 20:53:40.598229 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.598154    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"]
Sep 09 20:53:40.605563 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:40.605537    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lhnnc"]
Sep 09 20:53:41.492921 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:41.492884    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb37bbad-ae79-41c6-ada9-81e2becc083e" path="/var/lib/kubelet/pods/bb37bbad-ae79-41c6-ada9-81e2becc083e/volumes"
Sep 09 20:53:57.589432 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:53:57.589383    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"]
Sep 09 20:54:10.752667 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.752625    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"]
Sep 09 20:54:10.753205 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.752996    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb37bbad-ae79-41c6-ada9-81e2becc083e" containerName="main"
Sep 09 20:54:10.753205 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.753008    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb37bbad-ae79-41c6-ada9-81e2becc083e" containerName="main"
Sep 09 20:54:10.753205 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.753068    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb37bbad-ae79-41c6-ada9-81e2becc083e" containerName="main"
Sep 09 20:54:10.755919 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.755899    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.759806 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.759777    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-7s76d\""
Sep 09 20:54:10.759806 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.759792    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 09 20:54:10.760004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.759778    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 09 20:54:10.760004 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.759943    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 09 20:54:10.765668 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.765642    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"]
Sep 09 20:54:10.802832 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.802789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfscj\" (UniqueName: \"kubernetes.io/projected/bd582421-a66d-4bc7-860a-526d03082e9e-kube-api-access-gfscj\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.803028 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.802857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.803028 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.802944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-home\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.803028 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.802982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.803253 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.803069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582421-a66d-4bc7-860a-526d03082e9e-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.803253 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.803124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-dshm\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904256 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904256 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-home\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904519 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904519 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582421-a66d-4bc7-860a-526d03082e9e-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904519 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-dshm\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904687 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfscj\" (UniqueName: \"kubernetes.io/projected/bd582421-a66d-4bc7-860a-526d03082e9e-kube-api-access-gfscj\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904687 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904636    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-home\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904687 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.904848 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.904764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.906967 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.906940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-dshm\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.907257 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.907238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582421-a66d-4bc7-860a-526d03082e9e-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:10.912985 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:10.912961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfscj\" (UniqueName: \"kubernetes.io/projected/bd582421-a66d-4bc7-860a-526d03082e9e-kube-api-access-gfscj\") pod \"roll-v2-kserve-d4dbf9d67-mwl86\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:11.068296 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:11.068204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:11.347060 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:11.347025    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"]
Sep 09 20:54:11.348283 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:54:11.348250    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd582421_a66d_4bc7_860a_526d03082e9e.slice/crio-cae5bce48cbc346729e26a1fcd8df5f9377bbeae730c6d561793d70d727be2dc WatchSource:0}: Error finding container cae5bce48cbc346729e26a1fcd8df5f9377bbeae730c6d561793d70d727be2dc: Status 404 returned error can't find the container with id cae5bce48cbc346729e26a1fcd8df5f9377bbeae730c6d561793d70d727be2dc
Sep 09 20:54:11.732431 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:11.732388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86" event={"ID":"bd582421-a66d-4bc7-860a-526d03082e9e","Type":"ContainerStarted","Data":"f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5"}
Sep 09 20:54:11.732602 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:11.732448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86" event={"ID":"bd582421-a66d-4bc7-860a-526d03082e9e","Type":"ContainerStarted","Data":"cae5bce48cbc346729e26a1fcd8df5f9377bbeae730c6d561793d70d727be2dc"}
Sep 09 20:54:11.758074 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:11.758021    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86" podStartSLOduration=1.758004976 podStartE2EDuration="1.758004976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:11.75600143 +0000 UTC m=+2576.791472590" watchObservedRunningTime="2026-09-09 20:54:11.758004976 +0000 UTC m=+2576.793476135"
Sep 09 20:54:12.069290 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.069188    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:12.610036 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.609973    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm" podUID="535e2a14-bb32-4952-9a09-124e2d573025" containerName="main" containerID="cri-o://a4094d2ae03f116b0b05c17f094a30b224af6d452168ac85a1b4597c92cbc1a3" gracePeriod=45
Sep 09 20:54:12.739988 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.739955    2566 generic.go:356] "Generic (PLEG): container finished" podID="535e2a14-bb32-4952-9a09-124e2d573025" containerID="a4094d2ae03f116b0b05c17f094a30b224af6d452168ac85a1b4597c92cbc1a3" exitCode=0
Sep 09 20:54:12.740176 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.740025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm" event={"ID":"535e2a14-bb32-4952-9a09-124e2d573025","Type":"ContainerDied","Data":"a4094d2ae03f116b0b05c17f094a30b224af6d452168ac85a1b4597c92cbc1a3"}
Sep 09 20:54:12.867857 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.867781    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:54:12.923936 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.923903    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-dshm\") pod \"535e2a14-bb32-4952-9a09-124e2d573025\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") "
Sep 09 20:54:12.924155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.923966    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e2a14-bb32-4952-9a09-124e2d573025-tls-certs\") pod \"535e2a14-bb32-4952-9a09-124e2d573025\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") "
Sep 09 20:54:12.924155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx4mm\" (UniqueName: \"kubernetes.io/projected/535e2a14-bb32-4952-9a09-124e2d573025-kube-api-access-qx4mm\") pod \"535e2a14-bb32-4952-9a09-124e2d573025\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") "
Sep 09 20:54:12.924155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-model-cache\") pod \"535e2a14-bb32-4952-9a09-124e2d573025\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") "
Sep 09 20:54:12.924155 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924126    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-home\") pod \"535e2a14-bb32-4952-9a09-124e2d573025\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") "
Sep 09 20:54:12.924392 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924174    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-tmp-dir\") pod \"535e2a14-bb32-4952-9a09-124e2d573025\" (UID: \"535e2a14-bb32-4952-9a09-124e2d573025\") "
Sep 09 20:54:12.924392 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924337    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-model-cache" (OuterVolumeSpecName: "model-cache") pod "535e2a14-bb32-4952-9a09-124e2d573025" (UID: "535e2a14-bb32-4952-9a09-124e2d573025"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:12.924534 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-home" (OuterVolumeSpecName: "home") pod "535e2a14-bb32-4952-9a09-124e2d573025" (UID: "535e2a14-bb32-4952-9a09-124e2d573025"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:12.924593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924533    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:54:12.924593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.924548    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "535e2a14-bb32-4952-9a09-124e2d573025" (UID: "535e2a14-bb32-4952-9a09-124e2d573025"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:12.926427 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.926391    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-dshm" (OuterVolumeSpecName: "dshm") pod "535e2a14-bb32-4952-9a09-124e2d573025" (UID: "535e2a14-bb32-4952-9a09-124e2d573025"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:12.926530 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.926495    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535e2a14-bb32-4952-9a09-124e2d573025-kube-api-access-qx4mm" (OuterVolumeSpecName: "kube-api-access-qx4mm") pod "535e2a14-bb32-4952-9a09-124e2d573025" (UID: "535e2a14-bb32-4952-9a09-124e2d573025"). InnerVolumeSpecName "kube-api-access-qx4mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:12.926530 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:12.926507    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535e2a14-bb32-4952-9a09-124e2d573025-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "535e2a14-bb32-4952-9a09-124e2d573025" (UID: "535e2a14-bb32-4952-9a09-124e2d573025"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:13.025749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.025716    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/535e2a14-bb32-4952-9a09-124e2d573025-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:54:13.025749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.025746    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx4mm\" (UniqueName: \"kubernetes.io/projected/535e2a14-bb32-4952-9a09-124e2d573025-kube-api-access-qx4mm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:54:13.025749 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.025756    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:54:13.025978 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.025765    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:54:13.025978 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.025774    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/535e2a14-bb32-4952-9a09-124e2d573025-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:54:13.746241 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.746207    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"
Sep 09 20:54:13.746428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.746250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm" event={"ID":"535e2a14-bb32-4952-9a09-124e2d573025","Type":"ContainerDied","Data":"b0d05d62a550a02d1023a2e19d2694f30a283a759a1aad4f3b74968004f63b13"}
Sep 09 20:54:13.746428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.746312    2566 scope.go:117] "RemoveContainer" containerID="a4094d2ae03f116b0b05c17f094a30b224af6d452168ac85a1b4597c92cbc1a3"
Sep 09 20:54:13.765793 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.765765    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"]
Sep 09 20:54:13.770367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:13.770336    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-tlccm"]
Sep 09 20:54:15.493826 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:15.493790    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="535e2a14-bb32-4952-9a09-124e2d573025" path="/var/lib/kubelet/pods/535e2a14-bb32-4952-9a09-124e2d573025/volumes"
Sep 09 20:54:21.068702 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:21.068646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:21.081444 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:21.081414    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:21.796975 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:21.796941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:54:41.362304 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.362217    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"]
Sep 09 20:54:41.362693 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.362594    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535e2a14-bb32-4952-9a09-124e2d573025" containerName="main"
Sep 09 20:54:41.362693 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.362605    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="535e2a14-bb32-4952-9a09-124e2d573025" containerName="main"
Sep 09 20:54:41.362693 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.362679    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="535e2a14-bb32-4952-9a09-124e2d573025" containerName="main"
Sep 09 20:54:41.365668 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.365649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.370177 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.370149    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 09 20:54:41.370177 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.370172    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 09 20:54:41.370396 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.370253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-crlf8\""
Sep 09 20:54:41.370396 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.370272    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 09 20:54:41.375939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.375916    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"]
Sep 09 20:54:41.478448 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.478407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.478448 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.478451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.478752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.478535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45xr\" (UniqueName: \"kubernetes.io/projected/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-kube-api-access-n45xr\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.478752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.478578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.478752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.478608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.478752 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.478624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580043 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580043 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45xr\" (UniqueName: \"kubernetes.io/projected/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-kube-api-access-n45xr\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580576 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580625 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580578    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.580625 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.580616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.582539 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.582520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.582816 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.582797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.589628 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.589594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45xr\" (UniqueName: \"kubernetes.io/projected/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-kube-api-access-n45xr\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.676771 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.676665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:41.952260 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:41.952230    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"]
Sep 09 20:54:41.954803 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:54:41.954769    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b45e305_f0de_46fc_a88d_7e9bfc64fbfd.slice/crio-9a56e0a9aea4e0864ca2df25a2fcd35acc76d61484133a028a3ef491a7abc6f5 WatchSource:0}: Error finding container 9a56e0a9aea4e0864ca2df25a2fcd35acc76d61484133a028a3ef491a7abc6f5: Status 404 returned error can't find the container with id 9a56e0a9aea4e0864ca2df25a2fcd35acc76d61484133a028a3ef491a7abc6f5
Sep 09 20:54:42.889115 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:42.889052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" event={"ID":"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd","Type":"ContainerStarted","Data":"ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c"}
Sep 09 20:54:42.889608 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:42.889138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" event={"ID":"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd","Type":"ContainerStarted","Data":"9a56e0a9aea4e0864ca2df25a2fcd35acc76d61484133a028a3ef491a7abc6f5"}
Sep 09 20:54:42.907460 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:42.907408    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" podStartSLOduration=1.907391461 podStartE2EDuration="1.907391461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:42.906264531 +0000 UTC m=+2607.941735695" watchObservedRunningTime="2026-09-09 20:54:42.907391461 +0000 UTC m=+2607.942862620"
Sep 09 20:54:43.677557 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:43.677503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:51.677712 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:51.677667    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:51.690360 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:51.690329    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:54:51.942735 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:54:51.942639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:55:52.159168 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:55:52.159126    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"]
Sep 09 20:56:07.180345 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.180214    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86" podUID="bd582421-a66d-4bc7-860a-526d03082e9e" containerName="main" containerID="cri-o://f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5" gracePeriod=45
Sep 09 20:56:07.433073 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.432998    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:56:07.568150 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568074    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-dshm\") pod \"bd582421-a66d-4bc7-860a-526d03082e9e\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") "
Sep 09 20:56:07.568359 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-home\") pod \"bd582421-a66d-4bc7-860a-526d03082e9e\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") "
Sep 09 20:56:07.568359 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568247    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-tmp-dir\") pod \"bd582421-a66d-4bc7-860a-526d03082e9e\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") "
Sep 09 20:56:07.568359 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-model-cache\") pod \"bd582421-a66d-4bc7-860a-526d03082e9e\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") "
Sep 09 20:56:07.568359 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568315    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582421-a66d-4bc7-860a-526d03082e9e-tls-certs\") pod \"bd582421-a66d-4bc7-860a-526d03082e9e\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") "
Sep 09 20:56:07.568561 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568368    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfscj\" (UniqueName: \"kubernetes.io/projected/bd582421-a66d-4bc7-860a-526d03082e9e-kube-api-access-gfscj\") pod \"bd582421-a66d-4bc7-860a-526d03082e9e\" (UID: \"bd582421-a66d-4bc7-860a-526d03082e9e\") "
Sep 09 20:56:07.568561 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd582421-a66d-4bc7-860a-526d03082e9e" (UID: "bd582421-a66d-4bc7-860a-526d03082e9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:07.568561 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-home" (OuterVolumeSpecName: "home") pod "bd582421-a66d-4bc7-860a-526d03082e9e" (UID: "bd582421-a66d-4bc7-860a-526d03082e9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:07.568669 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd582421-a66d-4bc7-860a-526d03082e9e" (UID: "bd582421-a66d-4bc7-860a-526d03082e9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:07.568732 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568704    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:56:07.568782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568730    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:56:07.568782 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.568745    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:56:07.570747 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.570713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd582421-a66d-4bc7-860a-526d03082e9e-kube-api-access-gfscj" (OuterVolumeSpecName: "kube-api-access-gfscj") pod "bd582421-a66d-4bc7-860a-526d03082e9e" (UID: "bd582421-a66d-4bc7-860a-526d03082e9e"). InnerVolumeSpecName "kube-api-access-gfscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:07.570747 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.570729    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-dshm" (OuterVolumeSpecName: "dshm") pod "bd582421-a66d-4bc7-860a-526d03082e9e" (UID: "bd582421-a66d-4bc7-860a-526d03082e9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:07.570917 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.570778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd582421-a66d-4bc7-860a-526d03082e9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd582421-a66d-4bc7-860a-526d03082e9e" (UID: "bd582421-a66d-4bc7-860a-526d03082e9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:07.670045 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.669997    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd582421-a66d-4bc7-860a-526d03082e9e-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:56:07.670045 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.670034    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfscj\" (UniqueName: \"kubernetes.io/projected/bd582421-a66d-4bc7-860a-526d03082e9e-kube-api-access-gfscj\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:56:07.670045 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:07.670050    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd582421-a66d-4bc7-860a-526d03082e9e-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:56:08.292504 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.292459    2566 generic.go:356] "Generic (PLEG): container finished" podID="bd582421-a66d-4bc7-860a-526d03082e9e" containerID="f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5" exitCode=0
Sep 09 20:56:08.292504 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.292499    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"
Sep 09 20:56:08.293042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.292525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86" event={"ID":"bd582421-a66d-4bc7-860a-526d03082e9e","Type":"ContainerDied","Data":"f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5"}
Sep 09 20:56:08.293042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.292577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86" event={"ID":"bd582421-a66d-4bc7-860a-526d03082e9e","Type":"ContainerDied","Data":"cae5bce48cbc346729e26a1fcd8df5f9377bbeae730c6d561793d70d727be2dc"}
Sep 09 20:56:08.293042 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.292605    2566 scope.go:117] "RemoveContainer" containerID="f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5"
Sep 09 20:56:08.302466 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.302438    2566 scope.go:117] "RemoveContainer" containerID="f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5"
Sep 09 20:56:08.302791 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:56:08.302767    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5\": container with ID starting with f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5 not found: ID does not exist" containerID="f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5"
Sep 09 20:56:08.302849 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.302799    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5"} err="failed to get container status \"f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5\": rpc error: code = NotFound desc = could not find container \"f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5\": container with ID starting with f7bbcaf0671a93c05137df239e77720985ab5af06498587ea25a248674e457b5 not found: ID does not exist"
Sep 09 20:56:08.318223 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.318186    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"]
Sep 09 20:56:08.324269 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:08.324234    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-mwl86"]
Sep 09 20:56:09.492511 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:09.492474    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd582421-a66d-4bc7-860a-526d03082e9e" path="/var/lib/kubelet/pods/bd582421-a66d-4bc7-860a-526d03082e9e/volumes"
Sep 09 20:56:26.650715 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.650668    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"]
Sep 09 20:56:26.651310 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.651232    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd582421-a66d-4bc7-860a-526d03082e9e" containerName="main"
Sep 09 20:56:26.651310 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.651250    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd582421-a66d-4bc7-860a-526d03082e9e" containerName="main"
Sep 09 20:56:26.651386 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.651345    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd582421-a66d-4bc7-860a-526d03082e9e" containerName="main"
Sep 09 20:56:26.654524 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.654505    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.658116 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.657959    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 09 20:56:26.658116 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.658012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 09 20:56:26.658116 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.658056    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 09 20:56:26.658348 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.658144    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-tx7sq\""
Sep 09 20:56:26.663438 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.663414    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"]
Sep 09 20:56:26.735612 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.735571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-model-cache\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.735808 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.735632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-dshm\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.735808 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.735697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.735808 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.735737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7ee44-0827-473c-a045-1d50f5873811-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.735808 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.735776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-home\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.735946 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.735812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjzm5\" (UniqueName: \"kubernetes.io/projected/0af7ee44-0827-473c-a045-1d50f5873811-kube-api-access-tjzm5\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.836797 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.836752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-home\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.836797 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.836806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjzm5\" (UniqueName: \"kubernetes.io/projected/0af7ee44-0827-473c-a045-1d50f5873811-kube-api-access-tjzm5\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.837135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.836927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-model-cache\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.837135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.837036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-dshm\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.837135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.837068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.837135 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.837134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7ee44-0827-473c-a045-1d50f5873811-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.837408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.837265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-home\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.837478 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.837460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-model-cache\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.837642 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.837615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.839684 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.839659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-dshm\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.839844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.839826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7ee44-0827-473c-a045-1d50f5873811-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.845578 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.845552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjzm5\" (UniqueName: \"kubernetes.io/projected/0af7ee44-0827-473c-a045-1d50f5873811-kube-api-access-tjzm5\") pod \"owner-v1-kserve-75579ff5bc-9tc26\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:26.966667 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:26.966580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:27.246487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:27.246456    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"]
Sep 09 20:56:27.249126 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:56:27.249072    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af7ee44_0827_473c_a045_1d50f5873811.slice/crio-006cdd85237106074575b3a68047fe3c6a7df3e12e70b157aece9608b7fdf3c2 WatchSource:0}: Error finding container 006cdd85237106074575b3a68047fe3c6a7df3e12e70b157aece9608b7fdf3c2: Status 404 returned error can't find the container with id 006cdd85237106074575b3a68047fe3c6a7df3e12e70b157aece9608b7fdf3c2
Sep 09 20:56:27.381596 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:27.381553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26" event={"ID":"0af7ee44-0827-473c-a045-1d50f5873811","Type":"ContainerStarted","Data":"a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309"}
Sep 09 20:56:27.381784 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:27.381625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26" event={"ID":"0af7ee44-0827-473c-a045-1d50f5873811","Type":"ContainerStarted","Data":"006cdd85237106074575b3a68047fe3c6a7df3e12e70b157aece9608b7fdf3c2"}
Sep 09 20:56:27.401337 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:27.401269    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26" podStartSLOduration=1.4012446060000001 podStartE2EDuration="1.401244606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:27.398958769 +0000 UTC m=+2712.434429930" watchObservedRunningTime="2026-09-09 20:56:27.401244606 +0000 UTC m=+2712.436715767"
Sep 09 20:56:27.967463 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:27.967420    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:36.966732 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:36.966685    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:36.980044 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:36.980014    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:56:37.444054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:56:37.444013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:57:27.471054 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:27.471010    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"]
Sep 09 20:57:42.494876 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.494779    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26" podUID="0af7ee44-0827-473c-a045-1d50f5873811" containerName="main" containerID="cri-o://a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309" gracePeriod=45
Sep 09 20:57:42.754254 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.754180    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:57:42.758114 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.758050    2566 generic.go:356] "Generic (PLEG): container finished" podID="0af7ee44-0827-473c-a045-1d50f5873811" containerID="a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309" exitCode=0
Sep 09 20:57:42.758283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.758114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26" event={"ID":"0af7ee44-0827-473c-a045-1d50f5873811","Type":"ContainerDied","Data":"a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309"}
Sep 09 20:57:42.758283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.758127    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"
Sep 09 20:57:42.758283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.758176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26" event={"ID":"0af7ee44-0827-473c-a045-1d50f5873811","Type":"ContainerDied","Data":"006cdd85237106074575b3a68047fe3c6a7df3e12e70b157aece9608b7fdf3c2"}
Sep 09 20:57:42.758283 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.758207    2566 scope.go:117] "RemoveContainer" containerID="a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309"
Sep 09 20:57:42.766979 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.766950    2566 scope.go:117] "RemoveContainer" containerID="a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309"
Sep 09 20:57:42.767353 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:57:42.767329    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309\": container with ID starting with a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309 not found: ID does not exist" containerID="a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309"
Sep 09 20:57:42.767445 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.767365    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309"} err="failed to get container status \"a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309\": rpc error: code = NotFound desc = could not find container \"a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309\": container with ID starting with a4fdca806496d478ccf555db1f3fa99b880c6f7bf648be5d0517dc67982ed309 not found: ID does not exist"
Sep 09 20:57:42.832593 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-home\") pod \"0af7ee44-0827-473c-a045-1d50f5873811\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") "
Sep 09 20:57:42.832844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832633    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-dshm\") pod \"0af7ee44-0827-473c-a045-1d50f5873811\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") "
Sep 09 20:57:42.832844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-tmp-dir\") pod \"0af7ee44-0827-473c-a045-1d50f5873811\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") "
Sep 09 20:57:42.832844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832675    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-model-cache\") pod \"0af7ee44-0827-473c-a045-1d50f5873811\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") "
Sep 09 20:57:42.832844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7ee44-0827-473c-a045-1d50f5873811-tls-certs\") pod \"0af7ee44-0827-473c-a045-1d50f5873811\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") "
Sep 09 20:57:42.832844 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832758    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjzm5\" (UniqueName: \"kubernetes.io/projected/0af7ee44-0827-473c-a045-1d50f5873811-kube-api-access-tjzm5\") pod \"0af7ee44-0827-473c-a045-1d50f5873811\" (UID: \"0af7ee44-0827-473c-a045-1d50f5873811\") "
Sep 09 20:57:42.833161 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-home" (OuterVolumeSpecName: "home") pod "0af7ee44-0827-473c-a045-1d50f5873811" (UID: "0af7ee44-0827-473c-a045-1d50f5873811"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:42.833161 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832953    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-model-cache" (OuterVolumeSpecName: "model-cache") pod "0af7ee44-0827-473c-a045-1d50f5873811" (UID: "0af7ee44-0827-473c-a045-1d50f5873811"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:42.833161 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.832967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0af7ee44-0827-473c-a045-1d50f5873811" (UID: "0af7ee44-0827-473c-a045-1d50f5873811"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:42.833161 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.833024    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:57:42.833161 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.833042    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:57:42.835243 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.835221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-dshm" (OuterVolumeSpecName: "dshm") pod "0af7ee44-0827-473c-a045-1d50f5873811" (UID: "0af7ee44-0827-473c-a045-1d50f5873811"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:42.835341 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.835320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af7ee44-0827-473c-a045-1d50f5873811-kube-api-access-tjzm5" (OuterVolumeSpecName: "kube-api-access-tjzm5") pod "0af7ee44-0827-473c-a045-1d50f5873811" (UID: "0af7ee44-0827-473c-a045-1d50f5873811"). InnerVolumeSpecName "kube-api-access-tjzm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:42.835398 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.835372    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af7ee44-0827-473c-a045-1d50f5873811-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0af7ee44-0827-473c-a045-1d50f5873811" (UID: "0af7ee44-0827-473c-a045-1d50f5873811"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:42.934367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.934330    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af7ee44-0827-473c-a045-1d50f5873811-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:57:42.934367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.934365    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjzm5\" (UniqueName: \"kubernetes.io/projected/0af7ee44-0827-473c-a045-1d50f5873811-kube-api-access-tjzm5\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:57:42.934367 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.934376    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:57:42.934621 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:42.934386    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af7ee44-0827-473c-a045-1d50f5873811-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:57:43.086209 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:43.086150    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"]
Sep 09 20:57:43.090830 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:43.090797    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-9tc26"]
Sep 09 20:57:43.494167 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:57:43.494130    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af7ee44-0827-473c-a045-1d50f5873811" path="/var/lib/kubelet/pods/0af7ee44-0827-473c-a045-1d50f5873811/volumes"
Sep 09 20:58:00.239665 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.239621    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"]
Sep 09 20:58:00.249926 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:58:00.249882    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:58:00.249926 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:58:00.249926 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:58:00.249926 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c" cmd=["/bin/sleep","15"]
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.249944    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" message="" err=<
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:58:00.249958    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" podUID="5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" containerName="main" containerID="cri-o://ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c"
Sep 09 20:58:00.250343 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.250000    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" podUID="5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" containerName="main" containerID="cri-o://ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c" gracePeriod=60
Sep 09 20:58:00.617586 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.617556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:58:00.693462 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-dshm\") pod \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") "
Sep 09 20:58:00.693462 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693464    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-home\") pod \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") "
Sep 09 20:58:00.693678 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693547    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n45xr\" (UniqueName: \"kubernetes.io/projected/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-kube-api-access-n45xr\") pod \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") "
Sep 09 20:58:00.693678 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-model-cache\") pod \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") "
Sep 09 20:58:00.693774 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tls-certs\") pod \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") "
Sep 09 20:58:00.693774 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tmp-dir\") pod \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\" (UID: \"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd\") "
Sep 09 20:58:00.693774 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693728    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-home" (OuterVolumeSpecName: "home") pod "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" (UID: "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:00.693928 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693901    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" (UID: "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:00.694014 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.693996    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:58:00.694076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.694011    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" (UID: "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:00.694076 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.694018    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:58:00.695941 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.695916    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-dshm" (OuterVolumeSpecName: "dshm") pod "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" (UID: "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:00.696068 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.695956    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" (UID: "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:00.696068 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.696033    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-kube-api-access-n45xr" (OuterVolumeSpecName: "kube-api-access-n45xr") pod "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" (UID: "5b45e305-f0de-46fc-a88d-7e9bfc64fbfd"). InnerVolumeSpecName "kube-api-access-n45xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:00.795408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.795307    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:58:00.795408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.795345    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:58:00.795408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.795357    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:58:00.795408 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.795370    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n45xr\" (UniqueName: \"kubernetes.io/projected/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd-kube-api-access-n45xr\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 20:58:00.849428 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.849383    2566 generic.go:356] "Generic (PLEG): container finished" podID="5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" containerID="ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c" exitCode=0
Sep 09 20:58:00.849611 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.849442    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"
Sep 09 20:58:00.849611 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.849473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" event={"ID":"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd","Type":"ContainerDied","Data":"ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c"}
Sep 09 20:58:00.849611 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.849534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z" event={"ID":"5b45e305-f0de-46fc-a88d-7e9bfc64fbfd","Type":"ContainerDied","Data":"9a56e0a9aea4e0864ca2df25a2fcd35acc76d61484133a028a3ef491a7abc6f5"}
Sep 09 20:58:00.849611 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.849557    2566 scope.go:117] "RemoveContainer" containerID="ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c"
Sep 09 20:58:00.859356 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.859336    2566 scope.go:117] "RemoveContainer" containerID="ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c"
Sep 09 20:58:00.859673 ip-10-0-131-185 kubenswrapper[2566]: E0909 20:58:00.859654    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c\": container with ID starting with ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c not found: ID does not exist" containerID="ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c"
Sep 09 20:58:00.859718 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.859684    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c"} err="failed to get container status \"ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c\": rpc error: code = NotFound desc = could not find container \"ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c\": container with ID starting with ad4a8802e76a9623261121fd8c434fcb1c3c21bd112d8e6d394174e2c40af69c not found: ID does not exist"
Sep 09 20:58:00.883539 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.883499    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"]
Sep 09 20:58:00.892803 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:00.892737    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z822z"]
Sep 09 20:58:01.492019 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:01.491984    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" path="/var/lib/kubelet/pods/5b45e305-f0de-46fc-a88d-7e9bfc64fbfd/volumes"
Sep 09 20:58:54.361939 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.361893    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"]
Sep 09 20:58:54.362487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.362317    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af7ee44-0827-473c-a045-1d50f5873811" containerName="main"
Sep 09 20:58:54.362487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.362335    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af7ee44-0827-473c-a045-1d50f5873811" containerName="main"
Sep 09 20:58:54.362487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.362349    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" containerName="main"
Sep 09 20:58:54.362487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.362354    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" containerName="main"
Sep 09 20:58:54.362487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.362418    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b45e305-f0de-46fc-a88d-7e9bfc64fbfd" containerName="main"
Sep 09 20:58:54.362487 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.362427    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af7ee44-0827-473c-a045-1d50f5873811" containerName="main"
Sep 09 20:58:54.365418 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.365399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.368339 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.368313    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 09 20:58:54.369271 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.369253    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-bnhqh\""
Sep 09 20:58:54.369415 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.369355    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 09 20:58:54.369415 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.369357    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 09 20:58:54.371437 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.371410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.371560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.371447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8744714-0a35-4a09-be70-d4965055d384-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.371560 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.371478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.371680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.371564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.371680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.371608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlw7l\" (UniqueName: \"kubernetes.io/projected/a8744714-0a35-4a09-be70-d4965055d384-kube-api-access-xlw7l\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.371680 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.371653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.376355 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.376330    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"]
Sep 09 20:58:54.472216 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472216 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8744714-0a35-4a09-be70-d4965055d384-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472470 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472470 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472470 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlw7l\" (UniqueName: \"kubernetes.io/projected/a8744714-0a35-4a09-be70-d4965055d384-kube-api-access-xlw7l\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472470 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472748 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472719    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472748 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.472916 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.472769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.474729 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.474708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.474987 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.474967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8744714-0a35-4a09-be70-d4965055d384-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.481970 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.481941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlw7l\" (UniqueName: \"kubernetes.io/projected/a8744714-0a35-4a09-be70-d4965055d384-kube-api-access-xlw7l\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.677324 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.677070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:58:54.960727 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:54.960698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"]
Sep 09 20:58:54.963492 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:58:54.963460    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8744714_0a35_4a09_be70_d4965055d384.slice/crio-2e7ae75142b1b8eae69e9a84e28f4f3ced168dbdf5e61a471c046329ea355d35 WatchSource:0}: Error finding container 2e7ae75142b1b8eae69e9a84e28f4f3ced168dbdf5e61a471c046329ea355d35: Status 404 returned error can't find the container with id 2e7ae75142b1b8eae69e9a84e28f4f3ced168dbdf5e61a471c046329ea355d35
Sep 09 20:58:55.107375 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:55.107335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" event={"ID":"a8744714-0a35-4a09-be70-d4965055d384","Type":"ContainerStarted","Data":"035854c8921a2be86545411f6674c3e40dc2827b5e85a3ec5ff1a49878746867"}
Sep 09 20:58:55.107554 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:55.107392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" event={"ID":"a8744714-0a35-4a09-be70-d4965055d384","Type":"ContainerStarted","Data":"2e7ae75142b1b8eae69e9a84e28f4f3ced168dbdf5e61a471c046329ea355d35"}
Sep 09 20:58:55.125828 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:55.125775    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" podStartSLOduration=1.125758263 podStartE2EDuration="1.125758263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:55.124444586 +0000 UTC m=+2860.159915791" watchObservedRunningTime="2026-09-09 20:58:55.125758263 +0000 UTC m=+2860.161229457"
Sep 09 20:58:55.678258 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:58:55.678219    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:59:04.678311 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:04.678266    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:59:04.692037 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:04.691996    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:59:05.164611 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:05.164574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 20:59:34.988692 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:34.988584    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"]
Sep 09 20:59:34.992895 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:34.992863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:34.996418 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:34.996384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 09 20:59:34.996656 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:34.996637    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-w52gm\""
Sep 09 20:59:35.002535 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.002495    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"]
Sep 09 20:59:35.128217 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.128174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833aefaa-c236-45cb-b567-360a2a7fa71d-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:35.128414 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.128223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlnk\" (UniqueName: \"kubernetes.io/projected/833aefaa-c236-45cb-b567-360a2a7fa71d-kube-api-access-5vlnk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:35.229040 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.228987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833aefaa-c236-45cb-b567-360a2a7fa71d-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:35.229040 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.229029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlnk\" (UniqueName: \"kubernetes.io/projected/833aefaa-c236-45cb-b567-360a2a7fa71d-kube-api-access-5vlnk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:35.231793 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.231765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833aefaa-c236-45cb-b567-360a2a7fa71d-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:35.237947 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.237905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlnk\" (UniqueName: \"kubernetes.io/projected/833aefaa-c236-45cb-b567-360a2a7fa71d-kube-api-access-5vlnk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:35.306512 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.306402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:35.584567 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:35.584539    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"]
Sep 09 20:59:35.586864 ip-10-0-131-185 kubenswrapper[2566]: W0909 20:59:35.586833    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833aefaa_c236_45cb_b567_360a2a7fa71d.slice/crio-63568fe1b2c1501b55d48b63983dc911aba40101f87494a6a96eb1e18c5ab1b8 WatchSource:0}: Error finding container 63568fe1b2c1501b55d48b63983dc911aba40101f87494a6a96eb1e18c5ab1b8: Status 404 returned error can't find the container with id 63568fe1b2c1501b55d48b63983dc911aba40101f87494a6a96eb1e18c5ab1b8
Sep 09 20:59:36.301729 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:36.301685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" event={"ID":"833aefaa-c236-45cb-b567-360a2a7fa71d","Type":"ContainerStarted","Data":"3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445"}
Sep 09 20:59:36.302266 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:36.301752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" event={"ID":"833aefaa-c236-45cb-b567-360a2a7fa71d","Type":"ContainerStarted","Data":"63568fe1b2c1501b55d48b63983dc911aba40101f87494a6a96eb1e18c5ab1b8"}
Sep 09 20:59:36.302266 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:36.301890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 20:59:36.320765 ip-10-0-131-185 kubenswrapper[2566]: I0909 20:59:36.320703    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" podStartSLOduration=2.320683791 podStartE2EDuration="2.320683791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:36.318972243 +0000 UTC m=+2901.354443404" watchObservedRunningTime="2026-09-09 20:59:36.320683791 +0000 UTC m=+2901.356154950"
Sep 09 21:00:07.308482 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:07.308449    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 21:00:09.292425 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:09.292383    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"]
Sep 09 21:00:17.308553 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:17.308510    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:24.314311 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:24.314265    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main" containerID="cri-o://3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445" gracePeriod=45
Sep 09 21:00:27.308819 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:27.308773    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:37.308613 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:37.308526    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:37.309031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:37.308641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 21:00:47.308075 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:47.308028    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:54.570822 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.570786    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 21:00:54.647182 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.647137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833aefaa-c236-45cb-b567-360a2a7fa71d-tls-certs\") pod \"833aefaa-c236-45cb-b567-360a2a7fa71d\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") "
Sep 09 21:00:54.647380 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.647198    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vlnk\" (UniqueName: \"kubernetes.io/projected/833aefaa-c236-45cb-b567-360a2a7fa71d-kube-api-access-5vlnk\") pod \"833aefaa-c236-45cb-b567-360a2a7fa71d\" (UID: \"833aefaa-c236-45cb-b567-360a2a7fa71d\") "
Sep 09 21:00:54.649669 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.649635    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833aefaa-c236-45cb-b567-360a2a7fa71d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "833aefaa-c236-45cb-b567-360a2a7fa71d" (UID: "833aefaa-c236-45cb-b567-360a2a7fa71d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:54.649791 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.649714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833aefaa-c236-45cb-b567-360a2a7fa71d-kube-api-access-5vlnk" (OuterVolumeSpecName: "kube-api-access-5vlnk") pod "833aefaa-c236-45cb-b567-360a2a7fa71d" (UID: "833aefaa-c236-45cb-b567-360a2a7fa71d"). InnerVolumeSpecName "kube-api-access-5vlnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:54.687424 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.687388    2566 generic.go:356] "Generic (PLEG): container finished" podID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerID="3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445" exitCode=0
Sep 09 21:00:54.687602 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.687464    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"
Sep 09 21:00:54.687602 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.687465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" event={"ID":"833aefaa-c236-45cb-b567-360a2a7fa71d","Type":"ContainerDied","Data":"3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445"}
Sep 09 21:00:54.687602 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.687565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz" event={"ID":"833aefaa-c236-45cb-b567-360a2a7fa71d","Type":"ContainerDied","Data":"63568fe1b2c1501b55d48b63983dc911aba40101f87494a6a96eb1e18c5ab1b8"}
Sep 09 21:00:54.687602 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.687595    2566 scope.go:117] "RemoveContainer" containerID="3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445"
Sep 09 21:00:54.697500 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.697477    2566 scope.go:117] "RemoveContainer" containerID="3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445"
Sep 09 21:00:54.697820 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:00:54.697802    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445\": container with ID starting with 3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445 not found: ID does not exist" containerID="3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445"
Sep 09 21:00:54.697868 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.697829    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445"} err="failed to get container status \"3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445\": rpc error: code = NotFound desc = could not find container \"3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445\": container with ID starting with 3c1a63e9e77b4427a6b1e97eb26c278de7562fd501d701c3c7bb621531377445 not found: ID does not exist"
Sep 09 21:00:54.709974 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.709938    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"]
Sep 09 21:00:54.714989 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.714953    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheflmlz"]
Sep 09 21:00:54.748797 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.748764    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/833aefaa-c236-45cb-b567-360a2a7fa71d-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:00:54.748797 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:54.748796    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlnk\" (UniqueName: \"kubernetes.io/projected/833aefaa-c236-45cb-b567-360a2a7fa71d-kube-api-access-5vlnk\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:00:55.491926 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:55.491878    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" path="/var/lib/kubelet/pods/833aefaa-c236-45cb-b567-360a2a7fa71d/volumes"
Sep 09 21:00:59.382894 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.382846    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"]
Sep 09 21:00:59.393716 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:00:59.393665    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:00:59.393716 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:00:59.393716 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 21:00:59.393716 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="035854c8921a2be86545411f6674c3e40dc2827b5e85a3ec5ff1a49878746867" cmd=["/bin/sleep","15"]
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.393743    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" message="" err=<
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:00:59.393761    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" podUID="a8744714-0a35-4a09-be70-d4965055d384" containerName="main" containerID="cri-o://035854c8921a2be86545411f6674c3e40dc2827b5e85a3ec5ff1a49878746867"
Sep 09 21:00:59.394052 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.393814    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" podUID="a8744714-0a35-4a09-be70-d4965055d384" containerName="main" containerID="cri-o://035854c8921a2be86545411f6674c3e40dc2827b5e85a3ec5ff1a49878746867" gracePeriod=60
Sep 09 21:00:59.713524 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.713485    2566 generic.go:356] "Generic (PLEG): container finished" podID="a8744714-0a35-4a09-be70-d4965055d384" containerID="035854c8921a2be86545411f6674c3e40dc2827b5e85a3ec5ff1a49878746867" exitCode=0
Sep 09 21:00:59.713703 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.713554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" event={"ID":"a8744714-0a35-4a09-be70-d4965055d384","Type":"ContainerDied","Data":"035854c8921a2be86545411f6674c3e40dc2827b5e85a3ec5ff1a49878746867"}
Sep 09 21:00:59.748229 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.748194    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 21:00:59.792973 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.792937    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-model-cache\") pod \"a8744714-0a35-4a09-be70-d4965055d384\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") "
Sep 09 21:00:59.792973 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.792988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-dshm\") pod \"a8744714-0a35-4a09-be70-d4965055d384\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") "
Sep 09 21:00:59.793283 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8744714-0a35-4a09-be70-d4965055d384-tls-certs\") pod \"a8744714-0a35-4a09-be70-d4965055d384\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") "
Sep 09 21:00:59.793283 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-home\") pod \"a8744714-0a35-4a09-be70-d4965055d384\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") "
Sep 09 21:00:59.793283 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793155    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlw7l\" (UniqueName: \"kubernetes.io/projected/a8744714-0a35-4a09-be70-d4965055d384-kube-api-access-xlw7l\") pod \"a8744714-0a35-4a09-be70-d4965055d384\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") "
Sep 09 21:00:59.793283 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793178    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-tmp-dir\") pod \"a8744714-0a35-4a09-be70-d4965055d384\" (UID: \"a8744714-0a35-4a09-be70-d4965055d384\") "
Sep 09 21:00:59.793283 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793261    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8744714-0a35-4a09-be70-d4965055d384" (UID: "a8744714-0a35-4a09-be70-d4965055d384"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:59.793604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793472    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:00:59.793604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793472    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-home" (OuterVolumeSpecName: "home") pod "a8744714-0a35-4a09-be70-d4965055d384" (UID: "a8744714-0a35-4a09-be70-d4965055d384"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:59.793604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.793520    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8744714-0a35-4a09-be70-d4965055d384" (UID: "a8744714-0a35-4a09-be70-d4965055d384"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:59.795506 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.795479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8744714-0a35-4a09-be70-d4965055d384-kube-api-access-xlw7l" (OuterVolumeSpecName: "kube-api-access-xlw7l") pod "a8744714-0a35-4a09-be70-d4965055d384" (UID: "a8744714-0a35-4a09-be70-d4965055d384"). InnerVolumeSpecName "kube-api-access-xlw7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:59.795630 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.795503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-dshm" (OuterVolumeSpecName: "dshm") pod "a8744714-0a35-4a09-be70-d4965055d384" (UID: "a8744714-0a35-4a09-be70-d4965055d384"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:59.795630 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.795506    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8744714-0a35-4a09-be70-d4965055d384-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8744714-0a35-4a09-be70-d4965055d384" (UID: "a8744714-0a35-4a09-be70-d4965055d384"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:59.894716 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.894677    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:00:59.894716 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.894708    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8744714-0a35-4a09-be70-d4965055d384-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:00:59.894716 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.894720    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:00:59.894716 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.894728    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlw7l\" (UniqueName: \"kubernetes.io/projected/a8744714-0a35-4a09-be70-d4965055d384-kube-api-access-xlw7l\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:00:59.894991 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:00:59.894737    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8744714-0a35-4a09-be70-d4965055d384-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:01:00.720350 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:01:00.720318    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"
Sep 09 21:01:00.720350 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:01:00.720338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl" event={"ID":"a8744714-0a35-4a09-be70-d4965055d384","Type":"ContainerDied","Data":"2e7ae75142b1b8eae69e9a84e28f4f3ced168dbdf5e61a471c046329ea355d35"}
Sep 09 21:01:00.720835 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:01:00.720420    2566 scope.go:117] "RemoveContainer" containerID="035854c8921a2be86545411f6674c3e40dc2827b5e85a3ec5ff1a49878746867"
Sep 09 21:01:00.744889 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:01:00.744844    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"]
Sep 09 21:01:00.746876 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:01:00.746845    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969228fl"]
Sep 09 21:01:01.491332 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:01:01.491293    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8744714-0a35-4a09-be70-d4965055d384" path="/var/lib/kubelet/pods/a8744714-0a35-4a09-be70-d4965055d384/volumes"
Sep 09 21:02:06.581657 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.581563    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"]
Sep 09 21:02:06.582156 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.581909    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8744714-0a35-4a09-be70-d4965055d384" containerName="main"
Sep 09 21:02:06.582156 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.581920    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8744714-0a35-4a09-be70-d4965055d384" containerName="main"
Sep 09 21:02:06.582156 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.581933    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main"
Sep 09 21:02:06.582156 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.581938    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main"
Sep 09 21:02:06.582156 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.581998    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8744714-0a35-4a09-be70-d4965055d384" containerName="main"
Sep 09 21:02:06.582156 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.582005    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="833aefaa-c236-45cb-b567-360a2a7fa71d" containerName="main"
Sep 09 21:02:06.585183 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.585158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.587587 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.587562    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 09 21:02:06.587839 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.587822    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-r7pwt\""
Sep 09 21:02:06.587839 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.587830    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 09 21:02:06.588111 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.588073    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:02:06.588161 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.588109    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 09 21:02:06.588161 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.588133    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 09 21:02:06.595603 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.595574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"]
Sep 09 21:02:06.686727 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.686689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.686727 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.686728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.686983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.686750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23e97e51-cabf-4ace-a942-2918ead3fb43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.686983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.686767    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fx8k\" (UniqueName: \"kubernetes.io/projected/23e97e51-cabf-4ace-a942-2918ead3fb43-kube-api-access-6fx8k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.686983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.686863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.686983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.686912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.686983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.686935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e97e51-cabf-4ace-a942-2918ead3fb43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.687257 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.687048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.788521 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e97e51-cabf-4ace-a942-2918ead3fb43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.788738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.788738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.788738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.788738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23e97e51-cabf-4ace-a942-2918ead3fb43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.788738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fx8k\" (UniqueName: \"kubernetes.io/projected/23e97e51-cabf-4ace-a942-2918ead3fb43-kube-api-access-6fx8k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.788738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.789022 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.788759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.789219 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.789185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.789371 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.789221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.789371 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.789255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.789371 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.789293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.789738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.789650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23e97e51-cabf-4ace-a942-2918ead3fb43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.791204 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.791182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.791345 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.791328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e97e51-cabf-4ace-a942-2918ead3fb43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.802718 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.802683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fx8k\" (UniqueName: \"kubernetes.io/projected/23e97e51-cabf-4ace-a942-2918ead3fb43-kube-api-access-6fx8k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.897105 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.897052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:06.908580 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.908542    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"]
Sep 09 21:02:06.912546 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.912522    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:06.915597 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.915567    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-g2tvm\""
Sep 09 21:02:06.923224 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:06.923176    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"]
Sep 09 21:02:07.092652 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.092610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df82w\" (UniqueName: \"kubernetes.io/projected/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-kube-api-access-df82w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:07.092652 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.092658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:07.184742 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.184709    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"]
Sep 09 21:02:07.186949 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:02:07.186916    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e97e51_cabf_4ace_a942_2918ead3fb43.slice/crio-9675b2e4d6ac34cba6870170e7c13f4ccec231bc90d688725d93865c32223bb2 WatchSource:0}: Error finding container 9675b2e4d6ac34cba6870170e7c13f4ccec231bc90d688725d93865c32223bb2: Status 404 returned error can't find the container with id 9675b2e4d6ac34cba6870170e7c13f4ccec231bc90d688725d93865c32223bb2
Sep 09 21:02:07.193483 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.193452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df82w\" (UniqueName: \"kubernetes.io/projected/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-kube-api-access-df82w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:07.193625 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.193496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:07.196120 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.196064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:07.202903 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.202873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df82w\" (UniqueName: \"kubernetes.io/projected/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-kube-api-access-df82w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:07.235321 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.235289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:07.513394 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:07.513368    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"]
Sep 09 21:02:07.515597 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:02:07.515566    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4cc073_fce0_4dce_b3c0_72e94b6ccc68.slice/crio-ffb6652d6fcb7dda19ebfcecbbbe4e476a1354deb160b23043ed6e214cc1611b WatchSource:0}: Error finding container ffb6652d6fcb7dda19ebfcecbbbe4e476a1354deb160b23043ed6e214cc1611b: Status 404 returned error can't find the container with id ffb6652d6fcb7dda19ebfcecbbbe4e476a1354deb160b23043ed6e214cc1611b
Sep 09 21:02:08.054187 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:08.054144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" event={"ID":"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68","Type":"ContainerStarted","Data":"85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4"}
Sep 09 21:02:08.054698 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:08.054377    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:08.054698 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:08.054428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" event={"ID":"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68","Type":"ContainerStarted","Data":"ffb6652d6fcb7dda19ebfcecbbbe4e476a1354deb160b23043ed6e214cc1611b"}
Sep 09 21:02:08.056065 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:08.056038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" event={"ID":"23e97e51-cabf-4ace-a942-2918ead3fb43","Type":"ContainerStarted","Data":"5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e"}
Sep 09 21:02:08.056229 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:08.056098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" event={"ID":"23e97e51-cabf-4ace-a942-2918ead3fb43","Type":"ContainerStarted","Data":"9675b2e4d6ac34cba6870170e7c13f4ccec231bc90d688725d93865c32223bb2"}
Sep 09 21:02:08.072441 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:08.072375    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" podStartSLOduration=2.072356913 podStartE2EDuration="2.072356913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:08.070227461 +0000 UTC m=+3053.105698634" watchObservedRunningTime="2026-09-09 21:02:08.072356913 +0000 UTC m=+3053.107828126"
Sep 09 21:02:11.072967 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:11.072898    2566 generic.go:356] "Generic (PLEG): container finished" podID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerID="5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e" exitCode=0
Sep 09 21:02:11.073374 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:11.072974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" event={"ID":"23e97e51-cabf-4ace-a942-2918ead3fb43","Type":"ContainerDied","Data":"5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e"}
Sep 09 21:02:12.080360 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:12.080317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" event={"ID":"23e97e51-cabf-4ace-a942-2918ead3fb43","Type":"ContainerStarted","Data":"6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee"}
Sep 09 21:02:12.106464 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:12.106396    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podStartSLOduration=2.893723771 podStartE2EDuration="6.106377119s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.212653348s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:12.102906332 +0000 UTC m=+3057.138377485" watchObservedRunningTime="2026-09-09 21:02:12.106377119 +0000 UTC m=+3057.141848279"
Sep 09 21:02:12.897382 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:12.897336    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:16.897841 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:16.897791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:02:16.899660 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:16.899623    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:02:26.898497 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:26.898444    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:02:36.897898 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:36.897847    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:02:39.062970 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:39.062930    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:02:46.898179 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:46.898073    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:02:56.898477 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:02:56.898430    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:03:06.897693 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:03:06.897638    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:03:16.897732 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:03:16.897683    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:03:26.898108 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:03:26.898037    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:03:36.907921 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:03:36.907820    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:03:36.916354 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:03:36.916326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:04:10.310743 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:10.310707    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"]
Sep 09 21:04:10.329475 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:10.329424    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"]
Sep 09 21:04:19.062427 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:19.062387    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:25.335256 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:25.335188    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" containerID="cri-o://6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee" gracePeriod=45
Sep 09 21:04:25.350880 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:25.350833    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main" containerID="cri-o://85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4" gracePeriod=45
Sep 09 21:04:25.908517 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:25.908469    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 09 21:04:26.018276 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.018246    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:04:26.148636 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148602    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-model-cache\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.148841 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148646    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-dshm\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.148841 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23e97e51-cabf-4ace-a942-2918ead3fb43-cabundle-cert\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.148841 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148725    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fx8k\" (UniqueName: \"kubernetes.io/projected/23e97e51-cabf-4ace-a942-2918ead3fb43-kube-api-access-6fx8k\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.148841 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148761    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-kserve-provision-location\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.149050 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148920    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-home\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.149050 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148967    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-tmp-dir\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.149050 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.149030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e97e51-cabf-4ace-a942-2918ead3fb43-tls-certs\") pod \"23e97e51-cabf-4ace-a942-2918ead3fb43\" (UID: \"23e97e51-cabf-4ace-a942-2918ead3fb43\") "
Sep 09 21:04:26.149245 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.148921    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-model-cache" (OuterVolumeSpecName: "model-cache") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:26.149245 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.149171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e97e51-cabf-4ace-a942-2918ead3fb43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:26.149379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.149340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-home" (OuterVolumeSpecName: "home") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:26.149623 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.149493    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.149729 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.149605    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.149729 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.149700    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23e97e51-cabf-4ace-a942-2918ead3fb43-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.151297 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.151269    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-dshm" (OuterVolumeSpecName: "dshm") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:26.151412 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.151350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e97e51-cabf-4ace-a942-2918ead3fb43-kube-api-access-6fx8k" (OuterVolumeSpecName: "kube-api-access-6fx8k") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "kube-api-access-6fx8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:26.151597 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.151579    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e97e51-cabf-4ace-a942-2918ead3fb43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:26.160539 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.160500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:26.179393 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.179350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23e97e51-cabf-4ace-a942-2918ead3fb43" (UID: "23e97e51-cabf-4ace-a942-2918ead3fb43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:26.250529 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.250489    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23e97e51-cabf-4ace-a942-2918ead3fb43-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.250529 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.250524    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.250529 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.250536    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fx8k\" (UniqueName: \"kubernetes.io/projected/23e97e51-cabf-4ace-a942-2918ead3fb43-kube-api-access-6fx8k\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.250810 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.250548    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.250810 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.250557    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23e97e51-cabf-4ace-a942-2918ead3fb43-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:26.764243 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.764208    2566 generic.go:356] "Generic (PLEG): container finished" podID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerID="6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee" exitCode=0
Sep 09 21:04:26.764678 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.764278    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"
Sep 09 21:04:26.764678 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.764296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" event={"ID":"23e97e51-cabf-4ace-a942-2918ead3fb43","Type":"ContainerDied","Data":"6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee"}
Sep 09 21:04:26.764678 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.764366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c" event={"ID":"23e97e51-cabf-4ace-a942-2918ead3fb43","Type":"ContainerDied","Data":"9675b2e4d6ac34cba6870170e7c13f4ccec231bc90d688725d93865c32223bb2"}
Sep 09 21:04:26.764678 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.764396    2566 scope.go:117] "RemoveContainer" containerID="6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee"
Sep 09 21:04:26.776336 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.776315    2566 scope.go:117] "RemoveContainer" containerID="5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e"
Sep 09 21:04:26.785359 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.785334    2566 scope.go:117] "RemoveContainer" containerID="6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee"
Sep 09 21:04:26.785708 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:04:26.785685    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee\": container with ID starting with 6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee not found: ID does not exist" containerID="6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee"
Sep 09 21:04:26.785756 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.785718    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee"} err="failed to get container status \"6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee\": rpc error: code = NotFound desc = could not find container \"6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee\": container with ID starting with 6e7fa656f857c09209a72424c1f690378987108edd5fc2ae84abbfad8eb4d9ee not found: ID does not exist"
Sep 09 21:04:26.785756 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.785737    2566 scope.go:117] "RemoveContainer" containerID="5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e"
Sep 09 21:04:26.785963 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:04:26.785941    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e\": container with ID starting with 5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e not found: ID does not exist" containerID="5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e"
Sep 09 21:04:26.786005 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.785970    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e"} err="failed to get container status \"5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e\": rpc error: code = NotFound desc = could not find container \"5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e\": container with ID starting with 5ca7b7a70dcb0757e81ad38ff13151e88a170842c03b40e4d7509808d0f3948e not found: ID does not exist"
Sep 09 21:04:26.789496 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.789472    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"]
Sep 09 21:04:26.793948 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:26.793916    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-8778c7fcc-9fk4c"]
Sep 09 21:04:27.492047 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:27.492008    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" path="/var/lib/kubelet/pods/23e97e51-cabf-4ace-a942-2918ead3fb43/volumes"
Sep 09 21:04:29.062775 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:29.062735    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:39.062234 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:39.062187    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:39.062701 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:39.062297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:04:49.062195 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:49.062155    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:55.610279 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.610250    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:04:55.698862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.698760    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df82w\" (UniqueName: \"kubernetes.io/projected/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-kube-api-access-df82w\") pod \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") "
Sep 09 21:04:55.698862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.698818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-tls-certs\") pod \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\" (UID: \"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68\") "
Sep 09 21:04:55.701077 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.701042    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-kube-api-access-df82w" (OuterVolumeSpecName: "kube-api-access-df82w") pod "2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" (UID: "2f4cc073-fce0-4dce-b3c0-72e94b6ccc68"). InnerVolumeSpecName "kube-api-access-df82w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:55.701274 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.701253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" (UID: "2f4cc073-fce0-4dce-b3c0-72e94b6ccc68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:55.799845 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.799791    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:55.799845 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.799831    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df82w\" (UniqueName: \"kubernetes.io/projected/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68-kube-api-access-df82w\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:04:55.907228 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.907194    2566 generic.go:356] "Generic (PLEG): container finished" podID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerID="85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4" exitCode=0
Sep 09 21:04:55.907413 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.907236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" event={"ID":"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68","Type":"ContainerDied","Data":"85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4"}
Sep 09 21:04:55.907413 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.907242    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"
Sep 09 21:04:55.907413 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.907274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv" event={"ID":"2f4cc073-fce0-4dce-b3c0-72e94b6ccc68","Type":"ContainerDied","Data":"ffb6652d6fcb7dda19ebfcecbbbe4e476a1354deb160b23043ed6e214cc1611b"}
Sep 09 21:04:55.907413 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.907295    2566 scope.go:117] "RemoveContainer" containerID="85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4"
Sep 09 21:04:55.922246 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.922220    2566 scope.go:117] "RemoveContainer" containerID="85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4"
Sep 09 21:04:55.922590 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:04:55.922570    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4\": container with ID starting with 85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4 not found: ID does not exist" containerID="85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4"
Sep 09 21:04:55.922643 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.922600    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4"} err="failed to get container status \"85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4\": rpc error: code = NotFound desc = could not find container \"85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4\": container with ID starting with 85b08f266563bf90df6d78145f40c156f7f269ea087b1e39f9181fdd03aacab4 not found: ID does not exist"
Sep 09 21:04:55.934861 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.934823    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"]
Sep 09 21:04:55.939491 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:55.939457    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schezd8xv"]
Sep 09 21:04:57.491149 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:04:57.491110    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" path="/var/lib/kubelet/pods/2f4cc073-fce0-4dce-b3c0-72e94b6ccc68/volumes"
Sep 09 21:05:06.460300 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460268    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"]
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460617    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main"
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460629    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main"
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460638    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main"
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460643    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main"
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460655    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="storage-initializer"
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460660    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="storage-initializer"
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460723    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4cc073-fce0-4dce-b3c0-72e94b6ccc68" containerName="main"
Sep 09 21:05:06.460731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.460729    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="23e97e51-cabf-4ace-a942-2918ead3fb43" containerName="main"
Sep 09 21:05:06.463918 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.463900    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.467007 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.466978    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 09 21:05:06.467180 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.467017    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 09 21:05:06.467180 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.467029    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-5qd6v\""
Sep 09 21:05:06.467180 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.467030    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 09 21:05:06.467180 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.467121    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:05:06.467180 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.467154    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 09 21:05:06.478704 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.478674    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"]
Sep 09 21:05:06.591968 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.591879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgcd\" (UniqueName: \"kubernetes.io/projected/b461fbd1-7364-4436-9954-b6492c922145-kube-api-access-7wgcd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.591968 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.591927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.591968 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.591951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b461fbd1-7364-4436-9954-b6492c922145-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.592244 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.591998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.592244 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.592048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.592244 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.592118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.592244 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.592185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b461fbd1-7364-4436-9954-b6492c922145-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.592244 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.592209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.689055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.689010    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"]
Sep 09 21:05:06.692384 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.692362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:06.692687 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.692661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgcd\" (UniqueName: \"kubernetes.io/projected/b461fbd1-7364-4436-9954-b6492c922145-kube-api-access-7wgcd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.692758 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.692725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.692813 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.692788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b461fbd1-7364-4436-9954-b6492c922145-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.692868 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.692820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.692868 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.692847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.692958 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.692882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.693040 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.693022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b461fbd1-7364-4436-9954-b6492c922145-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.693132 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.693093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.693575 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.693369    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.693575 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.693395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.693575 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.693504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.693801 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.693664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.694381 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.694353    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b461fbd1-7364-4436-9954-b6492c922145-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.694996 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.694971    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-lrztj\""
Sep 09 21:05:06.695545 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.695518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.695894 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.695873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b461fbd1-7364-4436-9954-b6492c922145-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.702362 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.702337    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"]
Sep 09 21:05:06.707535 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.707512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgcd\" (UniqueName: \"kubernetes.io/projected/b461fbd1-7364-4436-9954-b6492c922145-kube-api-access-7wgcd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.775281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.775232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:06.793630 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.793591    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:06.793774 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.793635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-kube-api-access-dc5nd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:06.894685 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.894644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:06.894685 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.894689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-kube-api-access-dc5nd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:06.897359 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.897336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:06.903507 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:06.903483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-kube-api-access-dc5nd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:07.030656 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.030565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:07.058165 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.058127    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"]
Sep 09 21:05:07.062828 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:05:07.062794    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb461fbd1_7364_4436_9954_b6492c922145.slice/crio-31833942296f93540fcbdb05446fc265457d9238a676578bf4a96b09be1095b5 WatchSource:0}: Error finding container 31833942296f93540fcbdb05446fc265457d9238a676578bf4a96b09be1095b5: Status 404 returned error can't find the container with id 31833942296f93540fcbdb05446fc265457d9238a676578bf4a96b09be1095b5
Sep 09 21:05:07.313003 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.312973    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"]
Sep 09 21:05:07.315289 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:05:07.315257    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fbe9b4_030e_4253_ad8a_e8c2bc8da0cc.slice/crio-5361ebb90e659ac79d3c033d0398364b64b320503c5b33b650c6ab8fb595525d WatchSource:0}: Error finding container 5361ebb90e659ac79d3c033d0398364b64b320503c5b33b650c6ab8fb595525d: Status 404 returned error can't find the container with id 5361ebb90e659ac79d3c033d0398364b64b320503c5b33b650c6ab8fb595525d
Sep 09 21:05:07.968293 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.968247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" event={"ID":"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc","Type":"ContainerStarted","Data":"3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2"}
Sep 09 21:05:07.968763 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.968405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:07.968763 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.968437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" event={"ID":"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc","Type":"ContainerStarted","Data":"5361ebb90e659ac79d3c033d0398364b64b320503c5b33b650c6ab8fb595525d"}
Sep 09 21:05:07.970049 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.970021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" event={"ID":"b461fbd1-7364-4436-9954-b6492c922145","Type":"ContainerStarted","Data":"80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2"}
Sep 09 21:05:07.970170 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.970073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" event={"ID":"b461fbd1-7364-4436-9954-b6492c922145","Type":"ContainerStarted","Data":"31833942296f93540fcbdb05446fc265457d9238a676578bf4a96b09be1095b5"}
Sep 09 21:05:07.986784 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:07.986730    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" podStartSLOduration=1.9867132939999999 podStartE2EDuration="1.986713294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:07.985048984 +0000 UTC m=+3233.020520148" watchObservedRunningTime="2026-09-09 21:05:07.986713294 +0000 UTC m=+3233.022184460"
Sep 09 21:05:10.987714 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:10.987679    2566 generic.go:356] "Generic (PLEG): container finished" podID="b461fbd1-7364-4436-9954-b6492c922145" containerID="80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2" exitCode=0
Sep 09 21:05:10.988168 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:10.987769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" event={"ID":"b461fbd1-7364-4436-9954-b6492c922145","Type":"ContainerDied","Data":"80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2"}
Sep 09 21:05:11.996223 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:11.996184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" event={"ID":"b461fbd1-7364-4436-9954-b6492c922145","Type":"ContainerStarted","Data":"f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1"}
Sep 09 21:05:12.029614 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:12.029556    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podStartSLOduration=2.891402362 podStartE2EDuration="6.029536277s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.138133915s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:12.027198005 +0000 UTC m=+3237.062669167" watchObservedRunningTime="2026-09-09 21:05:12.029536277 +0000 UTC m=+3237.065007438"
Sep 09 21:05:12.776057 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:12.776017    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:16.775908 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:16.775855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:05:16.777574 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:16.777540    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:05:26.776469 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:26.776418    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:05:36.775739 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:36.775681    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:05:38.977025 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:38.976992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:05:46.775895 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:46.775847    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:05:56.776027 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:05:56.775972    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:06:06.776760 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:06.776701    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:06:16.776301 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:16.776244    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:06:26.776560 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:26.776507    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:06:36.786279 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:36.786191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:06:36.794278 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:36.794242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:06:54.633872 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:54.633822    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"]
Sep 09 21:06:54.635765 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:54.635734    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"]
Sep 09 21:06:58.977109 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:06:58.977036    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:08.976506 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:08.976462    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:09.655251 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:09.655199    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main" containerID="cri-o://3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2" gracePeriod=45
Sep 09 21:07:09.660771 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:09.660710    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" containerID="cri-o://f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1" gracePeriod=45
Sep 09 21:07:09.787509 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:09.787462    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 09 21:07:10.233318 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.233287    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:07:10.308072 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.307971    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-kserve-provision-location\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308072 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-tmp-dir\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308072 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b461fbd1-7364-4436-9954-b6492c922145-cabundle-cert\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308072 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308063    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wgcd\" (UniqueName: \"kubernetes.io/projected/b461fbd1-7364-4436-9954-b6492c922145-kube-api-access-7wgcd\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308449 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308120    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-model-cache\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308449 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-dshm\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308449 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308205    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-home\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308449 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b461fbd1-7364-4436-9954-b6492c922145-tls-certs\") pod \"b461fbd1-7364-4436-9954-b6492c922145\" (UID: \"b461fbd1-7364-4436-9954-b6492c922145\") "
Sep 09 21:07:10.308641 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:10.308641 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b461fbd1-7364-4436-9954-b6492c922145-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:10.308641 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308535    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-model-cache" (OuterVolumeSpecName: "model-cache") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:10.308793 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308665    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.308793 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308686    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b461fbd1-7364-4436-9954-b6492c922145-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.308793 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308736    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.308923 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.308900    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-home" (OuterVolumeSpecName: "home") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:10.310621 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.310588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b461fbd1-7364-4436-9954-b6492c922145-kube-api-access-7wgcd" (OuterVolumeSpecName: "kube-api-access-7wgcd") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "kube-api-access-7wgcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:10.310736 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.310673    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-dshm" (OuterVolumeSpecName: "dshm") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:10.310897 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.310879    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b461fbd1-7364-4436-9954-b6492c922145-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:10.332523 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.332480    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b461fbd1-7364-4436-9954-b6492c922145" (UID: "b461fbd1-7364-4436-9954-b6492c922145"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:10.409444 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.409399    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wgcd\" (UniqueName: \"kubernetes.io/projected/b461fbd1-7364-4436-9954-b6492c922145-kube-api-access-7wgcd\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.409444 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.409437    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.409444 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.409447    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.409444 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.409454    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b461fbd1-7364-4436-9954-b6492c922145-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.409734 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.409464    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b461fbd1-7364-4436-9954-b6492c922145-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:10.598747 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.598648    2566 generic.go:356] "Generic (PLEG): container finished" podID="b461fbd1-7364-4436-9954-b6492c922145" containerID="f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1" exitCode=0
Sep 09 21:07:10.598747 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.598709    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"
Sep 09 21:07:10.598747 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.598727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" event={"ID":"b461fbd1-7364-4436-9954-b6492c922145","Type":"ContainerDied","Data":"f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1"}
Sep 09 21:07:10.599019 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.598780    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf" event={"ID":"b461fbd1-7364-4436-9954-b6492c922145","Type":"ContainerDied","Data":"31833942296f93540fcbdb05446fc265457d9238a676578bf4a96b09be1095b5"}
Sep 09 21:07:10.599019 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.598802    2566 scope.go:117] "RemoveContainer" containerID="f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1"
Sep 09 21:07:10.609353 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.609331    2566 scope.go:117] "RemoveContainer" containerID="80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2"
Sep 09 21:07:10.620650 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.620457    2566 scope.go:117] "RemoveContainer" containerID="f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1"
Sep 09 21:07:10.621004 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:07:10.620985    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1\": container with ID starting with f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1 not found: ID does not exist" containerID="f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1"
Sep 09 21:07:10.621054 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.621015    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1"} err="failed to get container status \"f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1\": rpc error: code = NotFound desc = could not find container \"f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1\": container with ID starting with f6c2d7cc2161dceaeeb7dcd40d88dbd3914786a135d67f728ad33f6212a291c1 not found: ID does not exist"
Sep 09 21:07:10.621054 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.621032    2566 scope.go:117] "RemoveContainer" containerID="80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2"
Sep 09 21:07:10.621360 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:07:10.621341    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2\": container with ID starting with 80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2 not found: ID does not exist" containerID="80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2"
Sep 09 21:07:10.621414 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.621364    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2"} err="failed to get container status \"80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2\": rpc error: code = NotFound desc = could not find container \"80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2\": container with ID starting with 80a71ed9668061578a5774f534ba6aa052d812c978c554c132c95d7b8e1509d2 not found: ID does not exist"
Sep 09 21:07:10.626163 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.626131    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"]
Sep 09 21:07:10.632877 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:10.632847    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-645465d786-xktsf"]
Sep 09 21:07:11.491774 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:11.491742    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b461fbd1-7364-4436-9954-b6492c922145" path="/var/lib/kubelet/pods/b461fbd1-7364-4436-9954-b6492c922145/volumes"
Sep 09 21:07:18.976444 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:18.976403    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:18.976926 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:18.976517    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:07:28.977060 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:28.977015    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:38.977041 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:38.976997    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:39.913906 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:39.913831    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:07:40.069948 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.069887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-kube-api-access-dc5nd\") pod \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") "
Sep 09 21:07:40.070386 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.069980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-tls-certs\") pod \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\" (UID: \"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc\") "
Sep 09 21:07:40.072304 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.072275    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" (UID: "99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:40.072359 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.072298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-kube-api-access-dc5nd" (OuterVolumeSpecName: "kube-api-access-dc5nd") pod "99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" (UID: "99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc"). InnerVolumeSpecName "kube-api-access-dc5nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:40.171540 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.171441    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:40.171540 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.171476    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc5nd\" (UniqueName: \"kubernetes.io/projected/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc-kube-api-access-dc5nd\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:07:40.739450 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.739409    2566 generic.go:356] "Generic (PLEG): container finished" podID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerID="3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2" exitCode=0
Sep 09 21:07:40.739450 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.739450    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"
Sep 09 21:07:40.739690 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.739487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" event={"ID":"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc","Type":"ContainerDied","Data":"3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2"}
Sep 09 21:07:40.739690 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.739538    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5" event={"ID":"99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc","Type":"ContainerDied","Data":"5361ebb90e659ac79d3c033d0398364b64b320503c5b33b650c6ab8fb595525d"}
Sep 09 21:07:40.739690 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.739561    2566 scope.go:117] "RemoveContainer" containerID="3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2"
Sep 09 21:07:40.748795 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.748772    2566 scope.go:117] "RemoveContainer" containerID="3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2"
Sep 09 21:07:40.749138 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:07:40.749117    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2\": container with ID starting with 3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2 not found: ID does not exist" containerID="3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2"
Sep 09 21:07:40.749186 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.749147    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2"} err="failed to get container status \"3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2\": rpc error: code = NotFound desc = could not find container \"3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2\": container with ID starting with 3fdf880c8fe2a21e4e66ccbf573312e1b88ce5edfe000b63da413ca72cf3d7a2 not found: ID does not exist"
Sep 09 21:07:40.762940 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.762896    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"]
Sep 09 21:07:40.767275 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:40.767243    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schem9lr5"]
Sep 09 21:07:41.491328 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:07:41.491290    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" path="/var/lib/kubelet/pods/99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc/volumes"
Sep 09 21:08:00.158368 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158328    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"]
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158682    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="storage-initializer"
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158695    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="storage-initializer"
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158711    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main"
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158717    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main"
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158728    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main"
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158733    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main"
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158791    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b461fbd1-7364-4436-9954-b6492c922145" containerName="main"
Sep 09 21:08:00.158977 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.158802    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99fbe9b4-030e-4253-ad8a-e8c2bc8da0cc" containerName="main"
Sep 09 21:08:00.161896 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.161872    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.166044 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.166015    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 09 21:08:00.166360 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.166344    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-rt4sg\""
Sep 09 21:08:00.166657 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.166642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 09 21:08:00.166954 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.166939    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 09 21:08:00.167246 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.167230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 09 21:08:00.184917 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.184882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"]
Sep 09 21:08:00.236106 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.236047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-home\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.236331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.236115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-dshm\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.236331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.236146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.236331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.236172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d010a-ed18-4c54-95da-00334a2bc6d0-tls-certs\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.236331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.236295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqxp\" (UniqueName: \"kubernetes.io/projected/966d010a-ed18-4c54-95da-00334a2bc6d0-kube-api-access-8tqxp\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.236525 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.236338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-model-cache\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.236525 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.236364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.337522 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqxp\" (UniqueName: \"kubernetes.io/projected/966d010a-ed18-4c54-95da-00334a2bc6d0-kube-api-access-8tqxp\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.337745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-model-cache\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.337745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.337745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337576    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-home\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.337745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-dshm\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.337745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.337745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d010a-ed18-4c54-95da-00334a2bc6d0-tls-certs\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.338071 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.337987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-model-cache\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.338071 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.338016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.338071 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.338057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.338276 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.338119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-home\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.340106 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.340060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-dshm\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.340287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.340269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d010a-ed18-4c54-95da-00334a2bc6d0-tls-certs\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.348332 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.348304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqxp\" (UniqueName: \"kubernetes.io/projected/966d010a-ed18-4c54-95da-00334a2bc6d0-kube-api-access-8tqxp\") pod \"e2e-del-blocked-svc-kserve-84c6c75669-cr2p4\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.450027 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.449937    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"]
Sep 09 21:08:00.453666 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.453646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.456992 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.456964    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-vlbz8\""
Sep 09 21:08:00.465446 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.465417    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"]
Sep 09 21:08:00.472375 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.472338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:00.539999 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.539926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrs5g\" (UniqueName: \"kubernetes.io/projected/b93544be-4c54-47bf-bf57-47cfa64f1165-kube-api-access-lrs5g\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.540724 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.540106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93544be-4c54-47bf-bf57-47cfa64f1165-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.641447 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.641397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrs5g\" (UniqueName: \"kubernetes.io/projected/b93544be-4c54-47bf-bf57-47cfa64f1165-kube-api-access-lrs5g\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.641651 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.641479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93544be-4c54-47bf-bf57-47cfa64f1165-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.644207 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.644185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93544be-4c54-47bf-bf57-47cfa64f1165-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.651351 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.651326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrs5g\" (UniqueName: \"kubernetes.io/projected/b93544be-4c54-47bf-bf57-47cfa64f1165-kube-api-access-lrs5g\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.759988 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.759961    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"]
Sep 09 21:08:00.762751 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:08:00.762712    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966d010a_ed18_4c54_95da_00334a2bc6d0.slice/crio-f3f52cb7031bb287aaff5bde4f7d2c45ee492b19072c45c4a85c7a57afd07ad6 WatchSource:0}: Error finding container f3f52cb7031bb287aaff5bde4f7d2c45ee492b19072c45c4a85c7a57afd07ad6: Status 404 returned error can't find the container with id f3f52cb7031bb287aaff5bde4f7d2c45ee492b19072c45c4a85c7a57afd07ad6
Sep 09 21:08:00.764707 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.764690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:00.836906 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:00.836868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4" event={"ID":"966d010a-ed18-4c54-95da-00334a2bc6d0","Type":"ContainerStarted","Data":"f3f52cb7031bb287aaff5bde4f7d2c45ee492b19072c45c4a85c7a57afd07ad6"}
Sep 09 21:08:01.052524 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:01.048344    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"]
Sep 09 21:08:01.053496 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:08:01.053447    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93544be_4c54_47bf_bf57_47cfa64f1165.slice/crio-0b08548c4b486fe495e08139b6e67aa19d2f9e23378c3d22f6a98d2ee05af211 WatchSource:0}: Error finding container 0b08548c4b486fe495e08139b6e67aa19d2f9e23378c3d22f6a98d2ee05af211: Status 404 returned error can't find the container with id 0b08548c4b486fe495e08139b6e67aa19d2f9e23378c3d22f6a98d2ee05af211
Sep 09 21:08:01.843591 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:01.843554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4" event={"ID":"966d010a-ed18-4c54-95da-00334a2bc6d0","Type":"ContainerStarted","Data":"909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc"}
Sep 09 21:08:01.845173 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:01.845146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" event={"ID":"b93544be-4c54-47bf-bf57-47cfa64f1165","Type":"ContainerStarted","Data":"fd3d7ab378436f995031462222e8ef96d72e3b6e2a9fa234dbf0b40677f87751"}
Sep 09 21:08:01.845320 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:01.845195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" event={"ID":"b93544be-4c54-47bf-bf57-47cfa64f1165","Type":"ContainerStarted","Data":"0b08548c4b486fe495e08139b6e67aa19d2f9e23378c3d22f6a98d2ee05af211"}
Sep 09 21:08:01.845320 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:01.845270    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:01.888360 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:01.888297    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" podStartSLOduration=1.8882768319999998 podStartE2EDuration="1.888276832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:01.887371467 +0000 UTC m=+3406.922842627" watchObservedRunningTime="2026-09-09 21:08:01.888276832 +0000 UTC m=+3406.923747992"
Sep 09 21:08:03.309179 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:03.309135    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"]
Sep 09 21:08:03.318959 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:03.318916    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"]
Sep 09 21:08:03.856907 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:03.856862    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4" podUID="966d010a-ed18-4c54-95da-00334a2bc6d0" containerName="storage-initializer" containerID="cri-o://909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc" gracePeriod=60
Sep 09 21:08:06.117970 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.117939    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:06.192562 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192461    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tqxp\" (UniqueName: \"kubernetes.io/projected/966d010a-ed18-4c54-95da-00334a2bc6d0-kube-api-access-8tqxp\") pod \"966d010a-ed18-4c54-95da-00334a2bc6d0\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") "
Sep 09 21:08:06.192562 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-kserve-provision-location\") pod \"966d010a-ed18-4c54-95da-00334a2bc6d0\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") "
Sep 09 21:08:06.192562 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-dshm\") pod \"966d010a-ed18-4c54-95da-00334a2bc6d0\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") "
Sep 09 21:08:06.192862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192585    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-home\") pod \"966d010a-ed18-4c54-95da-00334a2bc6d0\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") "
Sep 09 21:08:06.192862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192650    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-model-cache\") pod \"966d010a-ed18-4c54-95da-00334a2bc6d0\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") "
Sep 09 21:08:06.192862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-tmp-dir\") pod \"966d010a-ed18-4c54-95da-00334a2bc6d0\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") "
Sep 09 21:08:06.192862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192749    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d010a-ed18-4c54-95da-00334a2bc6d0-tls-certs\") pod \"966d010a-ed18-4c54-95da-00334a2bc6d0\" (UID: \"966d010a-ed18-4c54-95da-00334a2bc6d0\") "
Sep 09 21:08:06.193073 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192943    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-home" (OuterVolumeSpecName: "home") pod "966d010a-ed18-4c54-95da-00334a2bc6d0" (UID: "966d010a-ed18-4c54-95da-00334a2bc6d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:06.193073 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192929    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "966d010a-ed18-4c54-95da-00334a2bc6d0" (UID: "966d010a-ed18-4c54-95da-00334a2bc6d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:06.193073 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.192995    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "966d010a-ed18-4c54-95da-00334a2bc6d0" (UID: "966d010a-ed18-4c54-95da-00334a2bc6d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:06.195144 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.195113    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966d010a-ed18-4c54-95da-00334a2bc6d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "966d010a-ed18-4c54-95da-00334a2bc6d0" (UID: "966d010a-ed18-4c54-95da-00334a2bc6d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:06.195449 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.195420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-dshm" (OuterVolumeSpecName: "dshm") pod "966d010a-ed18-4c54-95da-00334a2bc6d0" (UID: "966d010a-ed18-4c54-95da-00334a2bc6d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:06.195552 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.195502    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966d010a-ed18-4c54-95da-00334a2bc6d0-kube-api-access-8tqxp" (OuterVolumeSpecName: "kube-api-access-8tqxp") pod "966d010a-ed18-4c54-95da-00334a2bc6d0" (UID: "966d010a-ed18-4c54-95da-00334a2bc6d0"). InnerVolumeSpecName "kube-api-access-8tqxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:06.279320 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.279272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "966d010a-ed18-4c54-95da-00334a2bc6d0" (UID: "966d010a-ed18-4c54-95da-00334a2bc6d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:06.293552 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.293521    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:06.293552 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.293551    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:06.293681 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.293559    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966d010a-ed18-4c54-95da-00334a2bc6d0-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:06.293681 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.293568    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqxp\" (UniqueName: \"kubernetes.io/projected/966d010a-ed18-4c54-95da-00334a2bc6d0-kube-api-access-8tqxp\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:06.293681 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.293578    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:06.293681 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.293586    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:06.293681 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.293595    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966d010a-ed18-4c54-95da-00334a2bc6d0-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:06.871736 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.871696    2566 generic.go:356] "Generic (PLEG): container finished" podID="966d010a-ed18-4c54-95da-00334a2bc6d0" containerID="909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc" exitCode=0
Sep 09 21:08:06.871923 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.871759    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"
Sep 09 21:08:06.871923 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.871770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4" event={"ID":"966d010a-ed18-4c54-95da-00334a2bc6d0","Type":"ContainerDied","Data":"909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc"}
Sep 09 21:08:06.871923 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.871825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4" event={"ID":"966d010a-ed18-4c54-95da-00334a2bc6d0","Type":"ContainerDied","Data":"f3f52cb7031bb287aaff5bde4f7d2c45ee492b19072c45c4a85c7a57afd07ad6"}
Sep 09 21:08:06.871923 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.871852    2566 scope.go:117] "RemoveContainer" containerID="909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc"
Sep 09 21:08:06.881622 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.881598    2566 scope.go:117] "RemoveContainer" containerID="909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc"
Sep 09 21:08:06.881932 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:08:06.881912    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc\": container with ID starting with 909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc not found: ID does not exist" containerID="909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc"
Sep 09 21:08:06.881982 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.881942    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc"} err="failed to get container status \"909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc\": rpc error: code = NotFound desc = could not find container \"909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc\": container with ID starting with 909bfba8adb827c7d34842d9f616d78aac4a02d4b7915fbdd9e96d465747a4dc not found: ID does not exist"
Sep 09 21:08:06.917469 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.917425    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"]
Sep 09 21:08:06.922005 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:06.921973    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-84c6c75669-cr2p4"]
Sep 09 21:08:07.491546 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:07.491506    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966d010a-ed18-4c54-95da-00334a2bc6d0" path="/var/lib/kubelet/pods/966d010a-ed18-4c54-95da-00334a2bc6d0/volumes"
Sep 09 21:08:18.878162 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:18.878070    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" podUID="b93544be-4c54-47bf-bf57-47cfa64f1165" containerName="main" containerID="cri-o://fd3d7ab378436f995031462222e8ef96d72e3b6e2a9fa234dbf0b40677f87751" gracePeriod=45
Sep 09 21:08:29.644371 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.644334    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"]
Sep 09 21:08:29.644894 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.644691    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966d010a-ed18-4c54-95da-00334a2bc6d0" containerName="storage-initializer"
Sep 09 21:08:29.644894 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.644702    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="966d010a-ed18-4c54-95da-00334a2bc6d0" containerName="storage-initializer"
Sep 09 21:08:29.644894 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.644756    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="966d010a-ed18-4c54-95da-00334a2bc6d0" containerName="storage-initializer"
Sep 09 21:08:29.647930 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.647906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:29.650733 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.650706    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 09 21:08:29.650886 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.650729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-ctg7s\""
Sep 09 21:08:29.651753 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.651728    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 09 21:08:29.651862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.651793    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 09 21:08:29.658052 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.658025    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"]
Sep 09 21:08:29.807398 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.807353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47p4\" (UniqueName: \"kubernetes.io/projected/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kube-api-access-p47p4\") pod \"e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:29.807594 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.807412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:29.908396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.908294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47p4\" (UniqueName: \"kubernetes.io/projected/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kube-api-access-p47p4\") pod \"e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:29.908396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.908360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:29.908737 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.908720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:29.918901 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.918873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47p4\" (UniqueName: \"kubernetes.io/projected/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kube-api-access-p47p4\") pod \"e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:29.959794 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:29.959749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:30.241545 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:30.241510    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"]
Sep 09 21:08:30.243368 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:08:30.243341    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8fba66_1ea8_441b_b95e_21f69a6d02c5.slice/crio-d362bc9e39cb2aa2f644cc8dad1a48f8229c05a8902d9bce4baca5002c3e73f2 WatchSource:0}: Error finding container d362bc9e39cb2aa2f644cc8dad1a48f8229c05a8902d9bce4baca5002c3e73f2: Status 404 returned error can't find the container with id d362bc9e39cb2aa2f644cc8dad1a48f8229c05a8902d9bce4baca5002c3e73f2
Sep 09 21:08:30.992945 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:30.992904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc" event={"ID":"ca8fba66-1ea8-441b-b95e-21f69a6d02c5","Type":"ContainerStarted","Data":"bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59"}
Sep 09 21:08:30.993365 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:30.992967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc" event={"ID":"ca8fba66-1ea8-441b-b95e-21f69a6d02c5","Type":"ContainerStarted","Data":"d362bc9e39cb2aa2f644cc8dad1a48f8229c05a8902d9bce4baca5002c3e73f2"}
Sep 09 21:08:32.611503 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:32.611460    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"]
Sep 09 21:08:33.004559 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:33.004450    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc" podUID="ca8fba66-1ea8-441b-b95e-21f69a6d02c5" containerName="storage-initializer" containerID="cri-o://bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59" gracePeriod=30
Sep 09 21:08:33.860821 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:33.860777    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" podUID="b93544be-4c54-47bf-bf57-47cfa64f1165" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:08:35.667824 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:35.667794    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:35.763399 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:35.763295    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p47p4\" (UniqueName: \"kubernetes.io/projected/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kube-api-access-p47p4\") pod \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") "
Sep 09 21:08:35.763578 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:35.763453    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kserve-provision-location\") pod \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\" (UID: \"ca8fba66-1ea8-441b-b95e-21f69a6d02c5\") "
Sep 09 21:08:35.765809 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:35.765782    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kube-api-access-p47p4" (OuterVolumeSpecName: "kube-api-access-p47p4") pod "ca8fba66-1ea8-441b-b95e-21f69a6d02c5" (UID: "ca8fba66-1ea8-441b-b95e-21f69a6d02c5"). InnerVolumeSpecName "kube-api-access-p47p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:35.839986 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:35.839938    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca8fba66-1ea8-441b-b95e-21f69a6d02c5" (UID: "ca8fba66-1ea8-441b-b95e-21f69a6d02c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:35.864210 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:35.864178    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:35.864210 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:35.864211    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47p4\" (UniqueName: \"kubernetes.io/projected/ca8fba66-1ea8-441b-b95e-21f69a6d02c5-kube-api-access-p47p4\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:36.020317 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.020221    2566 generic.go:356] "Generic (PLEG): container finished" podID="ca8fba66-1ea8-441b-b95e-21f69a6d02c5" containerID="bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59" exitCode=0
Sep 09 21:08:36.020317 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.020261    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"
Sep 09 21:08:36.020514 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.020310    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc" event={"ID":"ca8fba66-1ea8-441b-b95e-21f69a6d02c5","Type":"ContainerDied","Data":"bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59"}
Sep 09 21:08:36.020514 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.020364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc" event={"ID":"ca8fba66-1ea8-441b-b95e-21f69a6d02c5","Type":"ContainerDied","Data":"d362bc9e39cb2aa2f644cc8dad1a48f8229c05a8902d9bce4baca5002c3e73f2"}
Sep 09 21:08:36.020514 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.020389    2566 scope.go:117] "RemoveContainer" containerID="bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59"
Sep 09 21:08:36.029942 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.029919    2566 scope.go:117] "RemoveContainer" containerID="bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59"
Sep 09 21:08:36.030288 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:08:36.030264    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59\": container with ID starting with bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59 not found: ID does not exist" containerID="bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59"
Sep 09 21:08:36.030374 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.030295    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59"} err="failed to get container status \"bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59\": rpc error: code = NotFound desc = could not find container \"bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59\": container with ID starting with bdd8fa3a5c52facb0cf750f39b2ae43b5015a90d11ba4841eb2271699150da59 not found: ID does not exist"
Sep 09 21:08:36.061766 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.061735    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"]
Sep 09 21:08:36.067634 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:36.067599    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-79c85bd8bb-qh8mc"]
Sep 09 21:08:37.492289 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:37.492250    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca8fba66-1ea8-441b-b95e-21f69a6d02c5" path="/var/lib/kubelet/pods/ca8fba66-1ea8-441b-b95e-21f69a6d02c5/volumes"
Sep 09 21:08:40.606480 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.606436    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"]
Sep 09 21:08:40.606935 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.606879    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca8fba66-1ea8-441b-b95e-21f69a6d02c5" containerName="storage-initializer"
Sep 09 21:08:40.606935 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.606895    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca8fba66-1ea8-441b-b95e-21f69a6d02c5" containerName="storage-initializer"
Sep 09 21:08:40.607029 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.606987    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca8fba66-1ea8-441b-b95e-21f69a6d02c5" containerName="storage-initializer"
Sep 09 21:08:40.611609 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.611591    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.616680 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.616645    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-ww8kj\""
Sep 09 21:08:40.616875 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.616704    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 21:08:40.616875 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.616718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 21:08:40.616875 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.616740    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:08:40.616875 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.616710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 21:08:40.616875 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.616715    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 09 21:08:40.631675 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.631635    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"]
Sep 09 21:08:40.707165 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kserve-provision-location\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.707364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-home\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.707364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-model-cache\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.707364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-dshm\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.707364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7765f8a9-03cf-4fc8-94d9-fa326139aec9-cabundle-cert\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.707364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tmp-dir\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.707364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tls-certs\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.707570 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.707408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkj8\" (UniqueName: \"kubernetes.io/projected/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kube-api-access-lwkj8\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.808625 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tmp-dir\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.808625 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808617    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tls-certs\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.808890 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkj8\" (UniqueName: \"kubernetes.io/projected/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kube-api-access-lwkj8\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.808890 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kserve-provision-location\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.808890 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-home\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.808890 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-model-cache\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.809134 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-dshm\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.809134 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.808967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7765f8a9-03cf-4fc8-94d9-fa326139aec9-cabundle-cert\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.809134 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.809032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tmp-dir\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.809134 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.809073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-home\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.809360 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.809142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kserve-provision-location\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.809360 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.809267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-model-cache\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.809684 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.809661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7765f8a9-03cf-4fc8-94d9-fa326139aec9-cabundle-cert\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.811289 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.811258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-dshm\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.811564 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.811544    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tls-certs\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.825649 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.825610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkj8\" (UniqueName: \"kubernetes.io/projected/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kube-api-access-lwkj8\") pod \"stop-feature-test-kserve-6f85dbb559-4nxwv\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:40.921984 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:40.921940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:41.213901 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:41.213869    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"]
Sep 09 21:08:41.216416 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:08:41.216387    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7765f8a9_03cf_4fc8_94d9_fa326139aec9.slice/crio-e9774464de5313187ac1505b8beb289bbc1e126350aec809db628fefe34f3b62 WatchSource:0}: Error finding container e9774464de5313187ac1505b8beb289bbc1e126350aec809db628fefe34f3b62: Status 404 returned error can't find the container with id e9774464de5313187ac1505b8beb289bbc1e126350aec809db628fefe34f3b62
Sep 09 21:08:42.052374 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:42.052337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" event={"ID":"7765f8a9-03cf-4fc8-94d9-fa326139aec9","Type":"ContainerStarted","Data":"cf837cc9fd9d362fd8084f130c60e0049b202705573f05489f4fa8cdb36569df"}
Sep 09 21:08:42.052840 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:42.052396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" event={"ID":"7765f8a9-03cf-4fc8-94d9-fa326139aec9","Type":"ContainerStarted","Data":"e9774464de5313187ac1505b8beb289bbc1e126350aec809db628fefe34f3b62"}
Sep 09 21:08:43.859108 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:43.859037    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" podUID="b93544be-4c54-47bf-bf57-47cfa64f1165" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:08:44.063165 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:44.063039    2566 generic.go:356] "Generic (PLEG): container finished" podID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerID="cf837cc9fd9d362fd8084f130c60e0049b202705573f05489f4fa8cdb36569df" exitCode=0
Sep 09 21:08:44.063165 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:44.063119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" event={"ID":"7765f8a9-03cf-4fc8-94d9-fa326139aec9","Type":"ContainerDied","Data":"cf837cc9fd9d362fd8084f130c60e0049b202705573f05489f4fa8cdb36569df"}
Sep 09 21:08:45.070023 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:45.069986    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" event={"ID":"7765f8a9-03cf-4fc8-94d9-fa326139aec9","Type":"ContainerStarted","Data":"2865639b6b9d9094e37b4a9ee7e1b9adf3c47e4816b8039b4edd41ffe663b2ca"}
Sep 09 21:08:45.094500 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:45.094429    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podStartSLOduration=2.596298971 podStartE2EDuration="5.094404601s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.49810563s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:45.092950915 +0000 UTC m=+3450.128422076" watchObservedRunningTime="2026-09-09 21:08:45.094404601 +0000 UTC m=+3450.129875766"
Sep 09 21:08:45.922542 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:45.922496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:49.094217 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.094179    2566 generic.go:356] "Generic (PLEG): container finished" podID="b93544be-4c54-47bf-bf57-47cfa64f1165" containerID="fd3d7ab378436f995031462222e8ef96d72e3b6e2a9fa234dbf0b40677f87751" exitCode=0
Sep 09 21:08:49.094644 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.094259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" event={"ID":"b93544be-4c54-47bf-bf57-47cfa64f1165","Type":"ContainerDied","Data":"fd3d7ab378436f995031462222e8ef96d72e3b6e2a9fa234dbf0b40677f87751"}
Sep 09 21:08:49.139736 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.139708    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:49.187636 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.187534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93544be-4c54-47bf-bf57-47cfa64f1165-tls-certs\") pod \"b93544be-4c54-47bf-bf57-47cfa64f1165\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") "
Sep 09 21:08:49.187829 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.187657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrs5g\" (UniqueName: \"kubernetes.io/projected/b93544be-4c54-47bf-bf57-47cfa64f1165-kube-api-access-lrs5g\") pod \"b93544be-4c54-47bf-bf57-47cfa64f1165\" (UID: \"b93544be-4c54-47bf-bf57-47cfa64f1165\") "
Sep 09 21:08:49.190124 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.190059    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93544be-4c54-47bf-bf57-47cfa64f1165-kube-api-access-lrs5g" (OuterVolumeSpecName: "kube-api-access-lrs5g") pod "b93544be-4c54-47bf-bf57-47cfa64f1165" (UID: "b93544be-4c54-47bf-bf57-47cfa64f1165"). InnerVolumeSpecName "kube-api-access-lrs5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:49.190668 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.190640    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93544be-4c54-47bf-bf57-47cfa64f1165-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b93544be-4c54-47bf-bf57-47cfa64f1165" (UID: "b93544be-4c54-47bf-bf57-47cfa64f1165"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:49.289157 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.289112    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93544be-4c54-47bf-bf57-47cfa64f1165-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:49.289157 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:49.289161    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrs5g\" (UniqueName: \"kubernetes.io/projected/b93544be-4c54-47bf-bf57-47cfa64f1165-kube-api-access-lrs5g\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:08:50.100624 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:50.100587    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"
Sep 09 21:08:50.101072 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:50.100624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs" event={"ID":"b93544be-4c54-47bf-bf57-47cfa64f1165","Type":"ContainerDied","Data":"0b08548c4b486fe495e08139b6e67aa19d2f9e23378c3d22f6a98d2ee05af211"}
Sep 09 21:08:50.101072 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:50.100687    2566 scope.go:117] "RemoveContainer" containerID="fd3d7ab378436f995031462222e8ef96d72e3b6e2a9fa234dbf0b40677f87751"
Sep 09 21:08:50.126810 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:50.126775    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"]
Sep 09 21:08:50.135150 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:50.135119    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-zpqrs"]
Sep 09 21:08:50.922496 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:50.922447    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:08:50.924148 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:50.924112    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:08:51.492165 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:08:51.492126    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93544be-4c54-47bf-bf57-47cfa64f1165" path="/var/lib/kubelet/pods/b93544be-4c54-47bf-bf57-47cfa64f1165/volumes"
Sep 09 21:09:00.923064 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:09:00.923009    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:09:10.923102 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:09:10.923028    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:09:20.922961 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:09:20.922911    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:09:30.923273 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:09:30.923223    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:09:40.922980 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:09:40.922870    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:09:50.922865 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:09:50.922818    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:10:00.922777 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:00.922724    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:10:07.531388 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.531343    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"]
Sep 09 21:10:07.531845 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.531825    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93544be-4c54-47bf-bf57-47cfa64f1165" containerName="main"
Sep 09 21:10:07.531845 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.531843    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93544be-4c54-47bf-bf57-47cfa64f1165" containerName="main"
Sep 09 21:10:07.531921 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.531903    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93544be-4c54-47bf-bf57-47cfa64f1165" containerName="main"
Sep 09 21:10:07.534844 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.534826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:07.538695 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.538669    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-7qcx4\""
Sep 09 21:10:07.538872 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.538667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 09 21:10:07.538872 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.538669    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 09 21:10:07.538872 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.538673    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 09 21:10:07.538872 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.538707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-zqk49\""
Sep 09 21:10:07.546732 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.546690    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"]
Sep 09 21:10:07.703296 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.703262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9j5\" (UniqueName: \"kubernetes.io/projected/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-kube-api-access-wq9j5\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:07.703490 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.703326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:07.804465 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.804359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9j5\" (UniqueName: \"kubernetes.io/projected/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-kube-api-access-wq9j5\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:07.804465 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.804430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:07.807378 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.807348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:07.815706 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.815668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9j5\" (UniqueName: \"kubernetes.io/projected/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-kube-api-access-wq9j5\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:07.846831 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:07.846797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:08.133581 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:08.133512    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"]
Sep 09 21:10:08.136404 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:10:08.136372    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92b8736_9b4f_4ef0_ac27_bde7e8b718b8.slice/crio-ef6d08e196c89df457484bfbe1ba27ab17423cc8fafdc13bc06497c79f65754e WatchSource:0}: Error finding container ef6d08e196c89df457484bfbe1ba27ab17423cc8fafdc13bc06497c79f65754e: Status 404 returned error can't find the container with id ef6d08e196c89df457484bfbe1ba27ab17423cc8fafdc13bc06497c79f65754e
Sep 09 21:10:08.504996 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:08.504887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" event={"ID":"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8","Type":"ContainerStarted","Data":"0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4"}
Sep 09 21:10:08.504996 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:08.504945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" event={"ID":"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8","Type":"ContainerStarted","Data":"ef6d08e196c89df457484bfbe1ba27ab17423cc8fafdc13bc06497c79f65754e"}
Sep 09 21:10:08.505291 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:08.504995    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:08.524721 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:08.524663    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" podStartSLOduration=1.52464623 podStartE2EDuration="1.52464623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:08.523957182 +0000 UTC m=+3533.559428355" watchObservedRunningTime="2026-09-09 21:10:08.52464623 +0000 UTC m=+3533.560117390"
Sep 09 21:10:10.922570 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:10.922524    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:10:20.923192 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:20.923143    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:10:30.923221 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:30.923164    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:10:39.513416 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:39.513368    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:10:40.922722 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:40.922660    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 09 21:10:50.933356 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:50.933317    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:10:50.941610 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:50.941575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:10:53.136191 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:53.136156    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"]
Sep 09 21:10:59.679993 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.679951    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"]
Sep 09 21:10:59.684212 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.684184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.697203 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.697171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"]
Sep 09 21:10:59.755767 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.755726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-tmp-dir\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.755983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.755779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-home\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.755983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.755796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04fe358f-2bc9-491c-89b0-abb309fde374-cabundle-cert\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.755983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.755865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-model-cache\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.755983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.755907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-dshm\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.755983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.755949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2mq\" (UniqueName: \"kubernetes.io/projected/04fe358f-2bc9-491c-89b0-abb309fde374-kube-api-access-sb2mq\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.755983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.755983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe358f-2bc9-491c-89b0-abb309fde374-tls-certs\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.756313 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.756031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-kserve-provision-location\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.856645 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.856611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-tmp-dir\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.856840 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.856663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-home\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.856897 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.856828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04fe358f-2bc9-491c-89b0-abb309fde374-cabundle-cert\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.856897 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.856883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-model-cache\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857074 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-dshm\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857177 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-home\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857177 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-tmp-dir\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857177 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2mq\" (UniqueName: \"kubernetes.io/projected/04fe358f-2bc9-491c-89b0-abb309fde374-kube-api-access-sb2mq\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857346 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe358f-2bc9-491c-89b0-abb309fde374-tls-certs\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857346 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-kserve-provision-location\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857346 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-model-cache\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857529 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-kserve-provision-location\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.857669 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.857648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04fe358f-2bc9-491c-89b0-abb309fde374-cabundle-cert\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.859398 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.859378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-dshm\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.859790 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.859773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe358f-2bc9-491c-89b0-abb309fde374-tls-certs\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.868274 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.868245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2mq\" (UniqueName: \"kubernetes.io/projected/04fe358f-2bc9-491c-89b0-abb309fde374-kube-api-access-sb2mq\") pod \"stop-feature-test-kserve-6f85dbb559-gg5qq\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:10:59.995898 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:10:59.995644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:11:00.285918 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:00.284679    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"]
Sep 09 21:11:00.288206 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:11:00.288169    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fe358f_2bc9_491c_89b0_abb309fde374.slice/crio-eb3db2de01d23b47946d417f4ceeb1bbc7d4895c090978e1eb4124cce6d6d697 WatchSource:0}: Error finding container eb3db2de01d23b47946d417f4ceeb1bbc7d4895c090978e1eb4124cce6d6d697: Status 404 returned error can't find the container with id eb3db2de01d23b47946d417f4ceeb1bbc7d4895c090978e1eb4124cce6d6d697
Sep 09 21:11:00.757879 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:00.757835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" event={"ID":"04fe358f-2bc9-491c-89b0-abb309fde374","Type":"ContainerStarted","Data":"08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862"}
Sep 09 21:11:00.758439 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:00.757893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" event={"ID":"04fe358f-2bc9-491c-89b0-abb309fde374","Type":"ContainerStarted","Data":"eb3db2de01d23b47946d417f4ceeb1bbc7d4895c090978e1eb4124cce6d6d697"}
Sep 09 21:11:02.769572 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:02.769532    2566 generic.go:356] "Generic (PLEG): container finished" podID="04fe358f-2bc9-491c-89b0-abb309fde374" containerID="08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862" exitCode=0
Sep 09 21:11:02.770002 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:02.769605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" event={"ID":"04fe358f-2bc9-491c-89b0-abb309fde374","Type":"ContainerDied","Data":"08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862"}
Sep 09 21:11:03.777700 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:03.777663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" event={"ID":"04fe358f-2bc9-491c-89b0-abb309fde374","Type":"ContainerStarted","Data":"d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e"}
Sep 09 21:11:03.807876 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:03.807805    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podStartSLOduration=2.462186745 podStartE2EDuration="4.807780235s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.34559349s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:03.806267503 +0000 UTC m=+3588.841738663" watchObservedRunningTime="2026-09-09 21:11:03.807780235 +0000 UTC m=+3588.843251398"
Sep 09 21:11:03.996400 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:03.996361    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:11:08.160280 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.160164    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main" containerID="cri-o://2865639b6b9d9094e37b4a9ee7e1b9adf3c47e4816b8039b4edd41ffe663b2ca" gracePeriod=45
Sep 09 21:11:08.810295 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.810258    2566 generic.go:356] "Generic (PLEG): container finished" podID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerID="2865639b6b9d9094e37b4a9ee7e1b9adf3c47e4816b8039b4edd41ffe663b2ca" exitCode=0
Sep 09 21:11:08.810295 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.810281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" event={"ID":"7765f8a9-03cf-4fc8-94d9-fa326139aec9","Type":"ContainerDied","Data":"2865639b6b9d9094e37b4a9ee7e1b9adf3c47e4816b8039b4edd41ffe663b2ca"}
Sep 09 21:11:08.919942 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.919908    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:11:08.946258 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.945816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwkj8\" (UniqueName: \"kubernetes.io/projected/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kube-api-access-lwkj8\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.946258 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.945869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-home\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.947026    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-model-cache\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.947125    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tmp-dir\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.947176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-dshm\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.947230    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7765f8a9-03cf-4fc8-94d9-fa326139aec9-cabundle-cert\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.947306    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kserve-provision-location\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.947328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tls-certs\") pod \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\" (UID: \"7765f8a9-03cf-4fc8-94d9-fa326139aec9\") "
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.947666    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-home" (OuterVolumeSpecName: "home") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.948046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-model-cache" (OuterVolumeSpecName: "model-cache") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:08.948637 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.948413    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7765f8a9-03cf-4fc8-94d9-fa326139aec9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:11:08.950303 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.950279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-dshm" (OuterVolumeSpecName: "dshm") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:08.951222 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.951180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kube-api-access-lwkj8" (OuterVolumeSpecName: "kube-api-access-lwkj8") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "kube-api-access-lwkj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:11:08.951544 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.951508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:11:08.982674 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.982305    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:08.983124 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:08.983069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7765f8a9-03cf-4fc8-94d9-fa326139aec9" (UID: "7765f8a9-03cf-4fc8-94d9-fa326139aec9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:09.049619 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049572    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7765f8a9-03cf-4fc8-94d9-fa326139aec9-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.049619 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049609    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.049619 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049619    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.049619 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049628    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwkj8\" (UniqueName: \"kubernetes.io/projected/7765f8a9-03cf-4fc8-94d9-fa326139aec9-kube-api-access-lwkj8\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.049917 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049638    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.049917 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049648    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.049917 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049656    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.049917 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.049663    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7765f8a9-03cf-4fc8-94d9-fa326139aec9-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:11:09.817031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.816929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv" event={"ID":"7765f8a9-03cf-4fc8-94d9-fa326139aec9","Type":"ContainerDied","Data":"e9774464de5313187ac1505b8beb289bbc1e126350aec809db628fefe34f3b62"}
Sep 09 21:11:09.817031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.816954    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"
Sep 09 21:11:09.817031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.816992    2566 scope.go:117] "RemoveContainer" containerID="2865639b6b9d9094e37b4a9ee7e1b9adf3c47e4816b8039b4edd41ffe663b2ca"
Sep 09 21:11:09.827165 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.827143    2566 scope.go:117] "RemoveContainer" containerID="cf837cc9fd9d362fd8084f130c60e0049b202705573f05489f4fa8cdb36569df"
Sep 09 21:11:09.844062 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.844027    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"]
Sep 09 21:11:09.848103 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.848054    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-4nxwv"]
Sep 09 21:11:09.996455 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.996415    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:11:09.998192 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:09.998144    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:11:11.493329 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:11.493295    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" path="/var/lib/kubelet/pods/7765f8a9-03cf-4fc8-94d9-fa326139aec9/volumes"
Sep 09 21:11:19.996790 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:19.996748    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:11:29.997157 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:29.997070    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:11:39.996654 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:39.996606    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:11:49.996898 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:49.996848    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:11:59.996569 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:11:59.996513    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:12:09.996568 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:09.996510    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:12:12.483676 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:12.483635    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"]
Sep 09 21:12:19.512279 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:19.512233    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:12:19.996658 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:19.996604    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:12:27.507454 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:27.507386    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main" containerID="cri-o://0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4" gracePeriod=45
Sep 09 21:12:29.512210 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:29.512165    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:12:29.997224 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:29.997173    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:12:39.512588 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:39.512492    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:12:39.513043 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:39.512613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:12:39.997287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:39.997235    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:12:49.512223 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:49.512184    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:12:49.997271 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:49.997226    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:12:57.771392 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:57.771310    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:12:57.870195 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:57.870156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-tls-certs\") pod \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") "
Sep 09 21:12:57.870352 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:57.870201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq9j5\" (UniqueName: \"kubernetes.io/projected/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-kube-api-access-wq9j5\") pod \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\" (UID: \"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8\") "
Sep 09 21:12:57.872722 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:57.872675    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" (UID: "b92b8736-9b4f-4ef0-ac27-bde7e8b718b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:57.872867 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:57.872685    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-kube-api-access-wq9j5" (OuterVolumeSpecName: "kube-api-access-wq9j5") pod "b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" (UID: "b92b8736-9b4f-4ef0-ac27-bde7e8b718b8"). InnerVolumeSpecName "kube-api-access-wq9j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:57.971242 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:57.971199    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:12:57.971242 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:57.971241    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9j5\" (UniqueName: \"kubernetes.io/projected/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8-kube-api-access-wq9j5\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:12:58.382041 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.382007    2566 generic.go:356] "Generic (PLEG): container finished" podID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerID="0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4" exitCode=0
Sep 09 21:12:58.382260 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.382068    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"
Sep 09 21:12:58.382260 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.382104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" event={"ID":"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8","Type":"ContainerDied","Data":"0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4"}
Sep 09 21:12:58.382260 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.382167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4" event={"ID":"b92b8736-9b4f-4ef0-ac27-bde7e8b718b8","Type":"ContainerDied","Data":"ef6d08e196c89df457484bfbe1ba27ab17423cc8fafdc13bc06497c79f65754e"}
Sep 09 21:12:58.382260 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.382190    2566 scope.go:117] "RemoveContainer" containerID="0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4"
Sep 09 21:12:58.391741 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.391719    2566 scope.go:117] "RemoveContainer" containerID="0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4"
Sep 09 21:12:58.392161 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:12:58.392133    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4\": container with ID starting with 0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4 not found: ID does not exist" containerID="0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4"
Sep 09 21:12:58.392267 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.392177    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4"} err="failed to get container status \"0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4\": rpc error: code = NotFound desc = could not find container \"0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4\": container with ID starting with 0bf3d91801dfd68850ac19e714eb8fc94a01e6ff8dfe37a6af0820c4862190d4 not found: ID does not exist"
Sep 09 21:12:58.408949 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.408910    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"]
Sep 09 21:12:58.413280 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:58.413249    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-9wcd4"]
Sep 09 21:12:59.492140 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:12:59.492105    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" path="/var/lib/kubelet/pods/b92b8736-9b4f-4ef0-ac27-bde7e8b718b8/volumes"
Sep 09 21:13:00.006517 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:00.006481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:13:00.015248 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:00.015213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:13:01.890162 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:01.890117    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"]
Sep 09 21:13:13.977259 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977222    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"]
Sep 09 21:13:13.977726 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977662    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main"
Sep 09 21:13:13.977726 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977680    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main"
Sep 09 21:13:13.977726 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977697    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="storage-initializer"
Sep 09 21:13:13.977726 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977705    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="storage-initializer"
Sep 09 21:13:13.977726 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977716    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main"
Sep 09 21:13:13.977726 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977724    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main"
Sep 09 21:13:13.978002 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977861    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92b8736-9b4f-4ef0-ac27-bde7e8b718b8" containerName="main"
Sep 09 21:13:13.978002 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.977877    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7765f8a9-03cf-4fc8-94d9-fa326139aec9" containerName="main"
Sep 09 21:13:13.981038 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.981019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:13.983773 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.983749    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 09 21:13:13.984060 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.983762    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:13:13.984203 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.984161    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 09 21:13:13.984286 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.984207    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-7pm8s\""
Sep 09 21:13:13.984895 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.984874    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 09 21:13:13.985001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.984874    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 09 21:13:13.997060 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:13.997030    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"]
Sep 09 21:13:14.022474 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.022665 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.022665 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.022665 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.022665 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.022806 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxpx\" (UniqueName: \"kubernetes.io/projected/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kube-api-access-qfxpx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.022806 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.022806 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.022758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.123701 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.123701 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxpx\" (UniqueName: \"kubernetes.io/projected/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kube-api-access-qfxpx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.123905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124388 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.124258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124388 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.124293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124528 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.124483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124686 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.124666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.124756 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.124716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.126371 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.126348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.126647 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.126629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.137400 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.137364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxpx\" (UniqueName: \"kubernetes.io/projected/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kube-api-access-qfxpx\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.292578 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.292461    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:14.587344 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:14.587258    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"]
Sep 09 21:13:14.591006 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:13:14.590967    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57c0d82_a326_49c1_9b8b_d7bf9420c6a6.slice/crio-b90e41484e25b9aae3f7b445bc0888091210d7f62bdded9983e35eca888ba19b WatchSource:0}: Error finding container b90e41484e25b9aae3f7b445bc0888091210d7f62bdded9983e35eca888ba19b: Status 404 returned error can't find the container with id b90e41484e25b9aae3f7b445bc0888091210d7f62bdded9983e35eca888ba19b
Sep 09 21:13:15.470926 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:15.470887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" event={"ID":"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6","Type":"ContainerStarted","Data":"7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441"}
Sep 09 21:13:15.471468 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:15.471163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" event={"ID":"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6","Type":"ContainerStarted","Data":"b90e41484e25b9aae3f7b445bc0888091210d7f62bdded9983e35eca888ba19b"}
Sep 09 21:13:16.921490 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:16.921405    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" containerID="cri-o://d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e" gracePeriod=45
Sep 09 21:13:17.008047 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.008003    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 09 21:13:17.482862 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.482823    2566 generic.go:356] "Generic (PLEG): container finished" podID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerID="7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441" exitCode=0
Sep 09 21:13:17.483033 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.482892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" event={"ID":"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6","Type":"ContainerDied","Data":"7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441"}
Sep 09 21:13:17.686000 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.685971    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:13:17.757287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757184    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-home\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.757287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04fe358f-2bc9-491c-89b0-abb309fde374-cabundle-cert\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.757287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757260    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-kserve-provision-location\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.757604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb2mq\" (UniqueName: \"kubernetes.io/projected/04fe358f-2bc9-491c-89b0-abb309fde374-kube-api-access-sb2mq\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.757604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-tmp-dir\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.757604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757353    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-dshm\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.757604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757412    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe358f-2bc9-491c-89b0-abb309fde374-tls-certs\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.757604 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757450    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-model-cache\") pod \"04fe358f-2bc9-491c-89b0-abb309fde374\" (UID: \"04fe358f-2bc9-491c-89b0-abb309fde374\") "
Sep 09 21:13:17.758007 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-home" (OuterVolumeSpecName: "home") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:17.758132 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.757988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fe358f-2bc9-491c-89b0-abb309fde374-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:17.758478 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.758450    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-model-cache" (OuterVolumeSpecName: "model-cache") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:17.760440 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.760410    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-dshm" (OuterVolumeSpecName: "dshm") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:17.760440 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.760426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe358f-2bc9-491c-89b0-abb309fde374-kube-api-access-sb2mq" (OuterVolumeSpecName: "kube-api-access-sb2mq") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "kube-api-access-sb2mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:17.760655 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.760632    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fe358f-2bc9-491c-89b0-abb309fde374-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:17.771229 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.771184    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:17.788894 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.788851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04fe358f-2bc9-491c-89b0-abb309fde374" (UID: "04fe358f-2bc9-491c-89b0-abb309fde374"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:17.859164 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859119    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2mq\" (UniqueName: \"kubernetes.io/projected/04fe358f-2bc9-491c-89b0-abb309fde374-kube-api-access-sb2mq\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:17.859164 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859166    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:17.859396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859181    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:17.859396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859197    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04fe358f-2bc9-491c-89b0-abb309fde374-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:17.859396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859208    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:17.859396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859218    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:17.859396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859229    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04fe358f-2bc9-491c-89b0-abb309fde374-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:17.859396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:17.859240    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04fe358f-2bc9-491c-89b0-abb309fde374-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:13:18.492627 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.492526    2566 generic.go:356] "Generic (PLEG): container finished" podID="04fe358f-2bc9-491c-89b0-abb309fde374" containerID="d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e" exitCode=0
Sep 09 21:13:18.492627 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.492618    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"
Sep 09 21:13:18.493195 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.492654    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" event={"ID":"04fe358f-2bc9-491c-89b0-abb309fde374","Type":"ContainerDied","Data":"d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e"}
Sep 09 21:13:18.493195 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.492720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq" event={"ID":"04fe358f-2bc9-491c-89b0-abb309fde374","Type":"ContainerDied","Data":"eb3db2de01d23b47946d417f4ceeb1bbc7d4895c090978e1eb4124cce6d6d697"}
Sep 09 21:13:18.493195 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.492751    2566 scope.go:117] "RemoveContainer" containerID="d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e"
Sep 09 21:13:18.495452 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.495423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" event={"ID":"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6","Type":"ContainerStarted","Data":"187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d"}
Sep 09 21:13:18.502611 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.502588    2566 scope.go:117] "RemoveContainer" containerID="08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862"
Sep 09 21:13:18.511828 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.511801    2566 scope.go:117] "RemoveContainer" containerID="d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e"
Sep 09 21:13:18.512261 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:13:18.512225    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e\": container with ID starting with d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e not found: ID does not exist" containerID="d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e"
Sep 09 21:13:18.512385 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.512275    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e"} err="failed to get container status \"d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e\": rpc error: code = NotFound desc = could not find container \"d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e\": container with ID starting with d4e94b00e2d0781492f3b33a1781717190a501e0d072905b27d4cbacf3eaaa4e not found: ID does not exist"
Sep 09 21:13:18.512385 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.512299    2566 scope.go:117] "RemoveContainer" containerID="08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862"
Sep 09 21:13:18.512610 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:13:18.512589    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862\": container with ID starting with 08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862 not found: ID does not exist" containerID="08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862"
Sep 09 21:13:18.512673 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.512620    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862"} err="failed to get container status \"08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862\": rpc error: code = NotFound desc = could not find container \"08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862\": container with ID starting with 08a94665e37e040a54bded6514d7a009773a4d5251c942ce9bcde95274c58862 not found: ID does not exist"
Sep 09 21:13:18.526602 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.526498    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podStartSLOduration=3.069029521 podStartE2EDuration="5.526478181s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.45744866s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:18.525317652 +0000 UTC m=+3723.560788824" watchObservedRunningTime="2026-09-09 21:13:18.526478181 +0000 UTC m=+3723.561949342"
Sep 09 21:13:18.543144 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.543071    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"]
Sep 09 21:13:18.550899 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:18.550859    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6f85dbb559-gg5qq"]
Sep 09 21:13:19.492437 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:19.492398    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" path="/var/lib/kubelet/pods/04fe358f-2bc9-491c-89b0-abb309fde374/volumes"
Sep 09 21:13:24.293339 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:24.293234    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:24.293847 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:24.293382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:13:24.295069 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:24.295037    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:13:34.293202 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:34.293153    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:13:44.293098 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:44.293029    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:13:54.293133 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:13:54.293056    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:14:04.293746 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:04.293686    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:14:14.293158 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:14.293048    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:14:17.479312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.479271    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"]
Sep 09 21:14:17.479798 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.479710    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main"
Sep 09 21:14:17.479798 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.479725    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main"
Sep 09 21:14:17.479798 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.479742    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="storage-initializer"
Sep 09 21:14:17.479798 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.479748    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="storage-initializer"
Sep 09 21:14:17.480031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.479829    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="04fe358f-2bc9-491c-89b0-abb309fde374" containerName="main"
Sep 09 21:14:17.484262 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.484235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.487423 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.487395    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 09 21:14:17.487735 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.487400    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 09 21:14:17.487861 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.487446    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:14:17.487930 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.487478    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 09 21:14:17.487930 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.487543    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-fhcr8\""
Sep 09 21:14:17.488045 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.487587    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 09 21:14:17.496612 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.496581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"]
Sep 09 21:14:17.614771 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-tmp-dir\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.614771 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-model-cache\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.615015 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-dshm\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.615015 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6w4\" (UniqueName: \"kubernetes.io/projected/ac426afe-63ad-40d6-89b7-c39923e26172-kube-api-access-tm6w4\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.615015 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac426afe-63ad-40d6-89b7-c39923e26172-tls-certs\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.615015 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac426afe-63ad-40d6-89b7-c39923e26172-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.615015 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-home\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.615015 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.614923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.715559 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-tmp-dir\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.715738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-model-cache\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.715738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-dshm\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.715738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6w4\" (UniqueName: \"kubernetes.io/projected/ac426afe-63ad-40d6-89b7-c39923e26172-kube-api-access-tm6w4\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.715738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac426afe-63ad-40d6-89b7-c39923e26172-tls-certs\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.715738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac426afe-63ad-40d6-89b7-c39923e26172-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.715738 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-home\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.716142 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.715904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.716142 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.716013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-tmp-dir\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.716142 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.716067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-home\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.716317 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.716298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.716392 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.716372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-model-cache\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.716619 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.716600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac426afe-63ad-40d6-89b7-c39923e26172-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.718166 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.718145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-dshm\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.718783 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.718764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac426afe-63ad-40d6-89b7-c39923e26172-tls-certs\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.727009 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.726976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6w4\" (UniqueName: \"kubernetes.io/projected/ac426afe-63ad-40d6-89b7-c39923e26172-kube-api-access-tm6w4\") pod \"lora-single-adapter-test-kserve-6d89cd65cf-fhvhr\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:17.797846 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:17.797755    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:18.086159 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:18.086013    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"]
Sep 09 21:14:18.090411 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:14:18.090373    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac426afe_63ad_40d6_89b7_c39923e26172.slice/crio-df466b5b19f31ff6db42ddbb564f92b2dde444fe8ce2a7c41ed1e267f5dfecff WatchSource:0}: Error finding container df466b5b19f31ff6db42ddbb564f92b2dde444fe8ce2a7c41ed1e267f5dfecff: Status 404 returned error can't find the container with id df466b5b19f31ff6db42ddbb564f92b2dde444fe8ce2a7c41ed1e267f5dfecff
Sep 09 21:14:18.805911 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:18.805869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" event={"ID":"ac426afe-63ad-40d6-89b7-c39923e26172","Type":"ContainerStarted","Data":"b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee"}
Sep 09 21:14:18.806337 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:18.805930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" event={"ID":"ac426afe-63ad-40d6-89b7-c39923e26172","Type":"ContainerStarted","Data":"df466b5b19f31ff6db42ddbb564f92b2dde444fe8ce2a7c41ed1e267f5dfecff"}
Sep 09 21:14:21.825507 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:21.825391    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac426afe-63ad-40d6-89b7-c39923e26172" containerID="b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee" exitCode=0
Sep 09 21:14:21.825507 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:21.825468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" event={"ID":"ac426afe-63ad-40d6-89b7-c39923e26172","Type":"ContainerDied","Data":"b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee"}
Sep 09 21:14:22.833504 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:22.833455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" event={"ID":"ac426afe-63ad-40d6-89b7-c39923e26172","Type":"ContainerStarted","Data":"e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd"}
Sep 09 21:14:22.859695 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:22.859628    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podStartSLOduration=2.667441394 podStartE2EDuration="5.859606786s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.192165392s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:22.858813666 +0000 UTC m=+3787.894284864" watchObservedRunningTime="2026-09-09 21:14:22.859606786 +0000 UTC m=+3787.895077946"
Sep 09 21:14:23.798408 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:23.798360    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:24.293641 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:24.293590    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:14:27.798956 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:27.798907    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:14:27.800755 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:27.800721    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:14:34.293670 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:34.293611    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:14:37.798534 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:37.798478    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:14:44.292893 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:44.292848    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:14:47.798997 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:47.798948    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:14:54.293134 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:54.293063    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:14:57.798819 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:14:57.798758    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:15:04.293823 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:04.293771    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:15:07.798255 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:07.798195    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:15:14.293752 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:14.293703    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 09 21:15:17.798390 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:17.798331    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:15:24.304197 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:24.304161    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:15:24.312307 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:24.312275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:15:27.798484 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:27.798421    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:15:34.485225 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:34.485185    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"]
Sep 09 21:15:37.798780 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:37.798684    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 09 21:15:47.808395 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:47.808357    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:15:47.816576 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:47.816543    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:15:49.509035 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:49.508967    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main" containerID="cri-o://187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d" gracePeriod=45
Sep 09 21:15:50.475509 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.475478    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:15:50.589700 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589603    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kserve-provision-location\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.589700 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfxpx\" (UniqueName: \"kubernetes.io/projected/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kube-api-access-qfxpx\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.589700 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-home\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.590312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589714    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-dshm\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.590312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tls-certs\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.590312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589779    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-cabundle-cert\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.590312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-model-cache\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.590312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.589855    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tmp-dir\") pod \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\" (UID: \"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6\") "
Sep 09 21:15:50.590312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.590158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.590312 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.590261    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.590677 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.590398    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:50.590677 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.590441    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-home" (OuterVolumeSpecName: "home") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.592587 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.592563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-dshm" (OuterVolumeSpecName: "dshm") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.592587 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.592574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kube-api-access-qfxpx" (OuterVolumeSpecName: "kube-api-access-qfxpx") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "kube-api-access-qfxpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:50.592858 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.592838    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:50.608321 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.608287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.610915 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.610891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" (UID: "e57c0d82-a326-49c1-9b8b-d7bf9420c6a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:50.691696 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.691661    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.691696 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.691693    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxpx\" (UniqueName: \"kubernetes.io/projected/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-kube-api-access-qfxpx\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.691696 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.691705    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.691928 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.691714    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.691928 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.691723    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.691928 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.691731    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:50.691928 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:50.691738    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:15:51.302272 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.302237    2566 generic.go:356] "Generic (PLEG): container finished" podID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerID="187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d" exitCode=0
Sep 09 21:15:51.302475 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.302295    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"
Sep 09 21:15:51.302475 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.302323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" event={"ID":"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6","Type":"ContainerDied","Data":"187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d"}
Sep 09 21:15:51.302475 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.302384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm" event={"ID":"e57c0d82-a326-49c1-9b8b-d7bf9420c6a6","Type":"ContainerDied","Data":"b90e41484e25b9aae3f7b445bc0888091210d7f62bdded9983e35eca888ba19b"}
Sep 09 21:15:51.302475 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.302413    2566 scope.go:117] "RemoveContainer" containerID="187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d"
Sep 09 21:15:51.311847 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.311808    2566 scope.go:117] "RemoveContainer" containerID="7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441"
Sep 09 21:15:51.319839 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.319822    2566 scope.go:117] "RemoveContainer" containerID="187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d"
Sep 09 21:15:51.320145 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:15:51.320127    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d\": container with ID starting with 187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d not found: ID does not exist" containerID="187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d"
Sep 09 21:15:51.320196 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.320153    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d"} err="failed to get container status \"187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d\": rpc error: code = NotFound desc = could not find container \"187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d\": container with ID starting with 187583a1e0cb7a38898059f9925968a505803626eaf386b8773c0e2da8f7e06d not found: ID does not exist"
Sep 09 21:15:51.320196 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.320170    2566 scope.go:117] "RemoveContainer" containerID="7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441"
Sep 09 21:15:51.320425 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:15:51.320406    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441\": container with ID starting with 7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441 not found: ID does not exist" containerID="7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441"
Sep 09 21:15:51.320464 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.320433    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441"} err="failed to get container status \"7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441\": rpc error: code = NotFound desc = could not find container \"7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441\": container with ID starting with 7c0256f1d6697283032490cffd414a6c1a72ed11a652709e5a0cb8c618952441 not found: ID does not exist"
Sep 09 21:15:51.329632 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.329606    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"]
Sep 09 21:15:51.334560 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.334536    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5758ccm"]
Sep 09 21:15:51.492246 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:15:51.492206    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" path="/var/lib/kubelet/pods/e57c0d82-a326-49c1-9b8b-d7bf9420c6a6/volumes"
Sep 09 21:16:11.571856 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:11.571809    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-6d89cd65cf-fhvhr_ac426afe-63ad-40d6-89b7-c39923e26172/storage-initializer/0.log"
Sep 09 21:16:11.582006 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:11.581987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-6d89cd65cf-fhvhr_ac426afe-63ad-40d6-89b7-c39923e26172/main/0.log"
Sep 09 21:16:13.041595 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:13.041551    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"]
Sep 09 21:16:28.064823 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.064758    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main" containerID="cri-o://e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd" gracePeriod=45
Sep 09 21:16:28.718683 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.718651    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:16:28.836953 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.836865    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-tmp-dir\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.836953 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.836912    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm6w4\" (UniqueName: \"kubernetes.io/projected/ac426afe-63ad-40d6-89b7-c39923e26172-kube-api-access-tm6w4\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.836953 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.836934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-kserve-provision-location\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.837281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.836969    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-home\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.837281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837000    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-dshm\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.837281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837055    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-model-cache\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.837281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac426afe-63ad-40d6-89b7-c39923e26172-tls-certs\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.837281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac426afe-63ad-40d6-89b7-c39923e26172-cabundle-cert\") pod \"ac426afe-63ad-40d6-89b7-c39923e26172\" (UID: \"ac426afe-63ad-40d6-89b7-c39923e26172\") "
Sep 09 21:16:28.837550 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837362    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:28.837550 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837435    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-home" (OuterVolumeSpecName: "home") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:28.837550 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837453    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:28.837857 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.837797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac426afe-63ad-40d6-89b7-c39923e26172-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:28.839579 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.839552    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac426afe-63ad-40d6-89b7-c39923e26172-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:28.839703 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.839604    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac426afe-63ad-40d6-89b7-c39923e26172-kube-api-access-tm6w4" (OuterVolumeSpecName: "kube-api-access-tm6w4") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "kube-api-access-tm6w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:28.839752 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.839727    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-dshm" (OuterVolumeSpecName: "dshm") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:28.850556 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.850521    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:28.859823 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.859803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac426afe-63ad-40d6-89b7-c39923e26172" (UID: "ac426afe-63ad-40d6-89b7-c39923e26172"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:28.937994 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.937969    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac426afe-63ad-40d6-89b7-c39923e26172-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:28.937994 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.937992    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac426afe-63ad-40d6-89b7-c39923e26172-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:28.938181 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.938003    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:28.938181 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.938012    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm6w4\" (UniqueName: \"kubernetes.io/projected/ac426afe-63ad-40d6-89b7-c39923e26172-kube-api-access-tm6w4\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:28.938181 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.938021    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:28.938181 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.938029    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:28.938181 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:28.938038    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac426afe-63ad-40d6-89b7-c39923e26172-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:16:29.495037 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.495002    2566 generic.go:356] "Generic (PLEG): container finished" podID="ac426afe-63ad-40d6-89b7-c39923e26172" containerID="e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd" exitCode=0
Sep 09 21:16:29.495535 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.495117    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"
Sep 09 21:16:29.495535 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.495233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" event={"ID":"ac426afe-63ad-40d6-89b7-c39923e26172","Type":"ContainerDied","Data":"e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd"}
Sep 09 21:16:29.495535 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.495285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr" event={"ID":"ac426afe-63ad-40d6-89b7-c39923e26172","Type":"ContainerDied","Data":"df466b5b19f31ff6db42ddbb564f92b2dde444fe8ce2a7c41ed1e267f5dfecff"}
Sep 09 21:16:29.495535 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.495319    2566 scope.go:117] "RemoveContainer" containerID="e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd"
Sep 09 21:16:29.505536 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.505517    2566 scope.go:117] "RemoveContainer" containerID="b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee"
Sep 09 21:16:29.513951 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.513935    2566 scope.go:117] "RemoveContainer" containerID="e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd"
Sep 09 21:16:29.514259 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:16:29.514239    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd\": container with ID starting with e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd not found: ID does not exist" containerID="e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd"
Sep 09 21:16:29.514313 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.514268    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd"} err="failed to get container status \"e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd\": rpc error: code = NotFound desc = could not find container \"e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd\": container with ID starting with e2c6f7a15dea553ddda280e625bd0a1505aba57e98b3ba487e9a6e09c93d79dd not found: ID does not exist"
Sep 09 21:16:29.514313 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.514286    2566 scope.go:117] "RemoveContainer" containerID="b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee"
Sep 09 21:16:29.514539 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:16:29.514520    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee\": container with ID starting with b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee not found: ID does not exist" containerID="b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee"
Sep 09 21:16:29.514595 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.514547    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee"} err="failed to get container status \"b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee\": rpc error: code = NotFound desc = could not find container \"b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee\": container with ID starting with b19c2e1fa4edf85c577d992478472c524d9705f8fbf9f33194ebf02504c908ee not found: ID does not exist"
Sep 09 21:16:29.522418 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.522394    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"]
Sep 09 21:16:29.526573 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:29.526552    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-6d89cd65cf-fhvhr"]
Sep 09 21:16:31.491899 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:31.491862    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" path="/var/lib/kubelet/pods/ac426afe-63ad-40d6-89b7-c39923e26172/volumes"
Sep 09 21:16:35.403459 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403422    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"]
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403784    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="storage-initializer"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403797    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="storage-initializer"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403807    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403814    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403829    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403835    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403844    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="storage-initializer"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403849    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="storage-initializer"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403906    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e57c0d82-a326-49c1-9b8b-d7bf9420c6a6" containerName="main"
Sep 09 21:16:35.404055 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.403913    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac426afe-63ad-40d6-89b7-c39923e26172" containerName="main"
Sep 09 21:16:35.407464 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.407421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.410506 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.410461    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 09 21:16:35.410627 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.410509    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 09 21:16:35.410627 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.410519    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-pkls4\""
Sep 09 21:16:35.410627 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.410514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 09 21:16:35.410627 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.410461    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:16:35.410627 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.410604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 09 21:16:35.411443 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.411425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-cvpbj\""
Sep 09 21:16:35.425442 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.425410    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"]
Sep 09 21:16:35.595023 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.594982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-dshm\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.595023 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.595019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-model-cache\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.595265 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.595145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45rg\" (UniqueName: \"kubernetes.io/projected/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kube-api-access-l45rg\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.595265 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.595172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-home\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.595265 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.595229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.595265 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.595251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.595452 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.595301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.595452 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.595418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.695942 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.695842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45rg\" (UniqueName: \"kubernetes.io/projected/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kube-api-access-l45rg\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.695942 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.695888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-home\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.695942 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.695919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.695945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.695969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-dshm\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696287 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-model-cache\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696546 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-home\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696546 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696546 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696546 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-model-cache\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.696816 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.696799    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.698598 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.698570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-dshm\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.698796 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.698778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.706581 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.706555    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45rg\" (UniqueName: \"kubernetes.io/projected/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kube-api-access-l45rg\") pod \"custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:35.717521 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:35.717504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:36.000160 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:36.000131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"]
Sep 09 21:16:36.003060 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:16:36.003031    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0fa525_91cf_4ef5_9dd7_f28aded0ed97.slice/crio-73066eccb838338724563d7e3ccde008d12cb3d67861f9d39832fcd1a86acc37 WatchSource:0}: Error finding container 73066eccb838338724563d7e3ccde008d12cb3d67861f9d39832fcd1a86acc37: Status 404 returned error can't find the container with id 73066eccb838338724563d7e3ccde008d12cb3d67861f9d39832fcd1a86acc37
Sep 09 21:16:36.005198 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:36.005180    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:16:36.527796 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:36.527759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerStarted","Data":"73066eccb838338724563d7e3ccde008d12cb3d67861f9d39832fcd1a86acc37"}
Sep 09 21:16:38.539134 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:38.539068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerStarted","Data":"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"}
Sep 09 21:16:38.539548 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:38.539160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:39.545391 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:39.545353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerStarted","Data":"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"}
Sep 09 21:16:41.558898 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:41.558862    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerID="82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd" exitCode=0
Sep 09 21:16:41.559410 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:41.558933    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerDied","Data":"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"}
Sep 09 21:16:42.567147 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:42.567111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerStarted","Data":"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"}
Sep 09 21:16:42.592711 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:42.592646    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podStartSLOduration=3.368323578 podStartE2EDuration="7.592622456s" totalImagesPullingTime="2.002299521s" totalInitContainerRuntime="2.221999357s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:42.589849336 +0000 UTC m=+3927.625320496" watchObservedRunningTime="2026-09-09 21:16:42.592622456 +0000 UTC m=+3927.628093617"
Sep 09 21:16:44.071743 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.071701    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"]
Sep 09 21:16:44.075254 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.075234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.077994 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.077971    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:16:44.078154 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.078010    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-r29c9\""
Sep 09 21:16:44.078154 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.078015    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 09 21:16:44.078154 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.077971    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 09 21:16:44.078154 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.078139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 09 21:16:44.078884 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.078867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 09 21:16:44.087176 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.087149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"]
Sep 09 21:16:44.171106 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-dshm\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.171106 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.171337 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e0c669-b34b-47fc-97e8-0dea8129b737-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.171337 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e0c669-b34b-47fc-97e8-0dea8129b737-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.171337 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-home\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.171337 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7l2m\" (UniqueName: \"kubernetes.io/projected/17e0c669-b34b-47fc-97e8-0dea8129b737-kube-api-access-c7l2m\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.171533 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-model-cache\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.171533 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.171419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272479 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-model-cache\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272479 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272486    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272722 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-dshm\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272722 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272722 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e0c669-b34b-47fc-97e8-0dea8129b737-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272722 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e0c669-b34b-47fc-97e8-0dea8129b737-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272929 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-home\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.272929 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7l2m\" (UniqueName: \"kubernetes.io/projected/17e0c669-b34b-47fc-97e8-0dea8129b737-kube-api-access-c7l2m\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.273127 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.272931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.273127 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.273058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.273320 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.273298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-model-cache\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.273382 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.273341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-home\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.273382 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.273338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e0c669-b34b-47fc-97e8-0dea8129b737-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.275266 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.275237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-dshm\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.275401 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.275384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e0c669-b34b-47fc-97e8-0dea8129b737-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.283251 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.283231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7l2m\" (UniqueName: \"kubernetes.io/projected/17e0c669-b34b-47fc-97e8-0dea8129b737-kube-api-access-c7l2m\") pod \"lora-multiple-adapters-test-kserve-769f46c5c6-9j96t\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.387189 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.387147    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:44.672284 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:44.672213    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"]
Sep 09 21:16:44.678311 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:16:44.678271    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e0c669_b34b_47fc_97e8_0dea8129b737.slice/crio-122a48894fea115efd42dfcf1b6eb032cd6031ba2c76c587ee9b7b9cc878570d WatchSource:0}: Error finding container 122a48894fea115efd42dfcf1b6eb032cd6031ba2c76c587ee9b7b9cc878570d: Status 404 returned error can't find the container with id 122a48894fea115efd42dfcf1b6eb032cd6031ba2c76c587ee9b7b9cc878570d
Sep 09 21:16:45.586101 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:45.586048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" event={"ID":"17e0c669-b34b-47fc-97e8-0dea8129b737","Type":"ContainerStarted","Data":"4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0"}
Sep 09 21:16:45.586545 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:45.586130    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" event={"ID":"17e0c669-b34b-47fc-97e8-0dea8129b737","Type":"ContainerStarted","Data":"122a48894fea115efd42dfcf1b6eb032cd6031ba2c76c587ee9b7b9cc878570d"}
Sep 09 21:16:45.718157 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:45.718077    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:45.718352 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:45.718198    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:16:45.719702 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:45.719661    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:16:48.605764 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:48.605730    2566 generic.go:356] "Generic (PLEG): container finished" podID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerID="4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0" exitCode=0
Sep 09 21:16:48.606162 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:48.605805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" event={"ID":"17e0c669-b34b-47fc-97e8-0dea8129b737","Type":"ContainerDied","Data":"4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0"}
Sep 09 21:16:49.613804 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:49.613766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" event={"ID":"17e0c669-b34b-47fc-97e8-0dea8129b737","Type":"ContainerStarted","Data":"2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395"}
Sep 09 21:16:49.640454 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:49.640381    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podStartSLOduration=2.366101263 podStartE2EDuration="5.640357008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.274255745s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:49.636636597 +0000 UTC m=+3934.672107758" watchObservedRunningTime="2026-09-09 21:16:49.640357008 +0000 UTC m=+3934.675828169"
Sep 09 21:16:50.387626 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:50.387579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:54.387456 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:54.387405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:16:54.389314 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:54.389278    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:16:55.718521 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:55.718463    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:16:55.736067 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:16:55.736031    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:17:04.387633 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:04.387573    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:17:05.719107 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:05.719033    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:17:14.388108 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:14.387991    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:17:15.718690 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:15.718611    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:17:24.387975 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:24.387921    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:17:25.719030 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:25.718982    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:17:34.388070 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:34.388022    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:17:35.718534 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:35.718478    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:17:44.388288 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:44.388238    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:17:45.719073 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:45.719023    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:17:54.388567 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:54.388521    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:17:55.718102 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:17:55.718015    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:18:04.388017 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:04.387973    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 09 21:18:05.719003 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:05.718949    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:18:14.398314 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:14.398278    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:18:14.406782 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:14.406748    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:18:15.717924 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:15.717875    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:18:25.718665 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:25.718586    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:18:35.718807 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:35.718754    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:18:45.718299 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:45.718196    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:18:55.718812 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:18:55.718759    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8001/health\": dial tcp 10.134.0.74:8001: connect: connection refused"
Sep 09 21:19:05.727541 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:05.727506    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:19:05.740721 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:05.740687    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:19:08.182001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:08.181961    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-769f46c5c6-9j96t_17e0c669-b34b-47fc-97e8-0dea8129b737/storage-initializer/0.log"
Sep 09 21:19:08.191612 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:08.191583    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-769f46c5c6-9j96t_17e0c669-b34b-47fc-97e8-0dea8129b737/main/0.log"
Sep 09 21:19:09.367775 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:09.367735    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"]
Sep 09 21:19:15.483500 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:15.483453    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"]
Sep 09 21:19:24.395585 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:24.395537    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main" containerID="cri-o://2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395" gracePeriod=45
Sep 09 21:19:25.053727 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.053699    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:19:25.191311 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-dshm\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191311 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191253    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-kserve-provision-location\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191311 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7l2m\" (UniqueName: \"kubernetes.io/projected/17e0c669-b34b-47fc-97e8-0dea8129b737-kube-api-access-c7l2m\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191311 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191314    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e0c669-b34b-47fc-97e8-0dea8129b737-tls-certs\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191676 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191333    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-tmp-dir\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191676 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191349    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e0c669-b34b-47fc-97e8-0dea8129b737-cabundle-cert\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191676 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-model-cache\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191676 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191406    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-home\") pod \"17e0c669-b34b-47fc-97e8-0dea8129b737\" (UID: \"17e0c669-b34b-47fc-97e8-0dea8129b737\") "
Sep 09 21:19:25.191895 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191690    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-model-cache" (OuterVolumeSpecName: "model-cache") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:25.191895 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.191815    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e0c669-b34b-47fc-97e8-0dea8129b737-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:25.192163 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.192129    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-home" (OuterVolumeSpecName: "home") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:25.194198 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.194049    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e0c669-b34b-47fc-97e8-0dea8129b737-kube-api-access-c7l2m" (OuterVolumeSpecName: "kube-api-access-c7l2m") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "kube-api-access-c7l2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:25.194316 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.194254    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-dshm" (OuterVolumeSpecName: "dshm") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:25.194316 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.194287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e0c669-b34b-47fc-97e8-0dea8129b737-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:25.204155 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.204127    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:25.222022 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.221986    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17e0c669-b34b-47fc-97e8-0dea8129b737" (UID: "17e0c669-b34b-47fc-97e8-0dea8129b737"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:25.293031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.292991    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e0c669-b34b-47fc-97e8-0dea8129b737-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.293031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.293025    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.293031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.293034    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.293031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.293043    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.293339 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.293051    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.293339 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.293062    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7l2m\" (UniqueName: \"kubernetes.io/projected/17e0c669-b34b-47fc-97e8-0dea8129b737-kube-api-access-c7l2m\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.293339 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.293070    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e0c669-b34b-47fc-97e8-0dea8129b737-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.293339 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.293078    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e0c669-b34b-47fc-97e8-0dea8129b737-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:25.454019 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.453923    2566 generic.go:356] "Generic (PLEG): container finished" podID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerID="2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395" exitCode=0
Sep 09 21:19:25.454019 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.453983    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"
Sep 09 21:19:25.454579 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.454011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" event={"ID":"17e0c669-b34b-47fc-97e8-0dea8129b737","Type":"ContainerDied","Data":"2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395"}
Sep 09 21:19:25.454579 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.454075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t" event={"ID":"17e0c669-b34b-47fc-97e8-0dea8129b737","Type":"ContainerDied","Data":"122a48894fea115efd42dfcf1b6eb032cd6031ba2c76c587ee9b7b9cc878570d"}
Sep 09 21:19:25.454579 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.454122    2566 scope.go:117] "RemoveContainer" containerID="2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395"
Sep 09 21:19:25.464014 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.463993    2566 scope.go:117] "RemoveContainer" containerID="4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0"
Sep 09 21:19:25.472693 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.472672    2566 scope.go:117] "RemoveContainer" containerID="2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395"
Sep 09 21:19:25.472996 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:19:25.472972    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395\": container with ID starting with 2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395 not found: ID does not exist" containerID="2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395"
Sep 09 21:19:25.473046 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.473006    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395"} err="failed to get container status \"2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395\": rpc error: code = NotFound desc = could not find container \"2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395\": container with ID starting with 2b3b107ff3fe0e1d741e870f30501c70f83bdb32aeaa31379f6ef32571b39395 not found: ID does not exist"
Sep 09 21:19:25.473046 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.473024    2566 scope.go:117] "RemoveContainer" containerID="4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0"
Sep 09 21:19:25.473326 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:19:25.473308    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0\": container with ID starting with 4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0 not found: ID does not exist" containerID="4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0"
Sep 09 21:19:25.473367 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.473332    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0"} err="failed to get container status \"4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0\": rpc error: code = NotFound desc = could not find container \"4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0\": container with ID starting with 4f88181ef7efc3694a5e1a6671ab13780c7c4de3c7d5a8ae85ab213b0cdda6c0 not found: ID does not exist"
Sep 09 21:19:25.479131 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.479105    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"]
Sep 09 21:19:25.484065 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.484039    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769f46c5c6-9j96t"]
Sep 09 21:19:25.491243 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:25.491217    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" path="/var/lib/kubelet/pods/17e0c669-b34b-47fc-97e8-0dea8129b737/volumes"
Sep 09 21:19:30.509597 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:30.509544    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main" containerID="cri-o://67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373" gracePeriod=45
Sep 09 21:19:31.128390 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.128350    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="llm-d-routing-sidecar" containerID="cri-o://bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae" gracePeriod=45
Sep 09 21:19:31.387463 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.387433    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:19:31.444356 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tmp-dir\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444571 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444441    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-cabundle-cert\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444571 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-home\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444571 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444507    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-dshm\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444571 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444530    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tls-certs\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444571 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444569    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kserve-provision-location\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444849 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444607    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l45rg\" (UniqueName: \"kubernetes.io/projected/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kube-api-access-l45rg\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444849 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-model-cache\") pod \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\" (UID: \"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97\") "
Sep 09 21:19:31.444956 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.444857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:31.445214 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.445186    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-home" (OuterVolumeSpecName: "home") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.445463 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.445438    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.447175 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.447148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-dshm" (OuterVolumeSpecName: "dshm") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.447273 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.447201    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:31.447494 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.447473    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kube-api-access-l45rg" (OuterVolumeSpecName: "kube-api-access-l45rg") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "kube-api-access-l45rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:31.459502 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.459466    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.470053 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.470012    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" (UID: "3b0fa525-91cf-4ef5-9dd7-f28aded0ed97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:31.485460 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.485415    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerID="67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373" exitCode=0
Sep 09 21:19:31.485460 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.485456    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerID="bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae" exitCode=0
Sep 09 21:19:31.485671 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.485535    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"
Sep 09 21:19:31.491292 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.491261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerDied","Data":"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"}
Sep 09 21:19:31.491441 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.491316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerDied","Data":"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"}
Sep 09 21:19:31.491441 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.491338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt" event={"ID":"3b0fa525-91cf-4ef5-9dd7-f28aded0ed97","Type":"ContainerDied","Data":"73066eccb838338724563d7e3ccde008d12cb3d67861f9d39832fcd1a86acc37"}
Sep 09 21:19:31.491441 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.491366    2566 scope.go:117] "RemoveContainer" containerID="67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"
Sep 09 21:19:31.500474 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.500455    2566 scope.go:117] "RemoveContainer" containerID="82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"
Sep 09 21:19:31.508520 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.508498    2566 scope.go:117] "RemoveContainer" containerID="bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"
Sep 09 21:19:31.514618 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.514590    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"]
Sep 09 21:19:31.518064 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.518040    2566 scope.go:117] "RemoveContainer" containerID="67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"
Sep 09 21:19:31.518492 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:19:31.518470    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373\": container with ID starting with 67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373 not found: ID does not exist" containerID="67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"
Sep 09 21:19:31.518565 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.518502    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"} err="failed to get container status \"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373\": rpc error: code = NotFound desc = could not find container \"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373\": container with ID starting with 67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373 not found: ID does not exist"
Sep 09 21:19:31.518565 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.518542    2566 scope.go:117] "RemoveContainer" containerID="82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"
Sep 09 21:19:31.518858 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:19:31.518838    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd\": container with ID starting with 82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd not found: ID does not exist" containerID="82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"
Sep 09 21:19:31.518927 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.518864    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"} err="failed to get container status \"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd\": rpc error: code = NotFound desc = could not find container \"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd\": container with ID starting with 82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd not found: ID does not exist"
Sep 09 21:19:31.518927 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.518881    2566 scope.go:117] "RemoveContainer" containerID="bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"
Sep 09 21:19:31.519169 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:19:31.519152    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae\": container with ID starting with bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae not found: ID does not exist" containerID="bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"
Sep 09 21:19:31.519232 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.519174    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"} err="failed to get container status \"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae\": rpc error: code = NotFound desc = could not find container \"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae\": container with ID starting with bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae not found: ID does not exist"
Sep 09 21:19:31.519232 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.519189    2566 scope.go:117] "RemoveContainer" containerID="67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"
Sep 09 21:19:31.519443 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.519423    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373"} err="failed to get container status \"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373\": rpc error: code = NotFound desc = could not find container \"67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373\": container with ID starting with 67fe1fd8417661f18b5e4231261fb0f2575f5e5d0a5ae1dfebbc3a1a2803e373 not found: ID does not exist"
Sep 09 21:19:31.519513 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.519446    2566 scope.go:117] "RemoveContainer" containerID="82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"
Sep 09 21:19:31.519732 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.519705    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd"} err="failed to get container status \"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd\": rpc error: code = NotFound desc = could not find container \"82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd\": container with ID starting with 82b5efbe4ec0b1c231a8a76e6f3f4edd6b7d724643df7dc6689e0175cb9457bd not found: ID does not exist"
Sep 09 21:19:31.519780 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.519733    2566 scope.go:117] "RemoveContainer" containerID="bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"
Sep 09 21:19:31.519994 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.519973    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae"} err="failed to get container status \"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae\": rpc error: code = NotFound desc = could not find container \"bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae\": container with ID starting with bbeed91141aa953fd068317f1bbbbb163824a5deb0c89ee83d70553aec8311ae not found: ID does not exist"
Sep 09 21:19:31.520040 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.520027    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-d98f8dfdd-kvpwt"]
Sep 09 21:19:31.545632 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545598    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.545632 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545630    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.545632 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545639    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.545899 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545647    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.545899 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545655    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.545899 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545662    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.545899 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545670    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:31.545899 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:31.545682    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l45rg\" (UniqueName: \"kubernetes.io/projected/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97-kube-api-access-l45rg\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:19:33.491288 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:33.491255    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" path="/var/lib/kubelet/pods/3b0fa525-91cf-4ef5-9dd7-f28aded0ed97/volumes"
Sep 09 21:19:35.377942 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.377908    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"]
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378359    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="llm-d-routing-sidecar"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378375    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="llm-d-routing-sidecar"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378386    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378392    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378401    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="storage-initializer"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378407    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="storage-initializer"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378413    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="storage-initializer"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378418    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="storage-initializer"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378424    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main"
Sep 09 21:19:35.378438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378429    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main"
Sep 09 21:19:35.378755 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378493    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="main"
Sep 09 21:19:35.378755 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378500    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e0c669-b34b-47fc-97e8-0dea8129b737" containerName="main"
Sep 09 21:19:35.378755 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.378509    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0fa525-91cf-4ef5-9dd7-f28aded0ed97" containerName="llm-d-routing-sidecar"
Sep 09 21:19:35.381593 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.381574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.385406 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.385382    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 09 21:19:35.385406 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.385383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-4bpx4\""
Sep 09 21:19:35.385623 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.385383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 09 21:19:35.385623 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.385383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-gm9cd\""
Sep 09 21:19:35.385623 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.385512    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 09 21:19:35.392252 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.392229    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"]
Sep 09 21:19:35.476843 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.476807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5xs\" (UniqueName: \"kubernetes.io/projected/1e74a258-cf66-4f55-b1c9-91e9edac8e40-kube-api-access-6t5xs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.477038 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.476878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e74a258-cf66-4f55-b1c9-91e9edac8e40-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.578199 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.578160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e74a258-cf66-4f55-b1c9-91e9edac8e40-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.578391 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.578218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5xs\" (UniqueName: \"kubernetes.io/projected/1e74a258-cf66-4f55-b1c9-91e9edac8e40-kube-api-access-6t5xs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.580890 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.580867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e74a258-cf66-4f55-b1c9-91e9edac8e40-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.588598 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.588569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5xs\" (UniqueName: \"kubernetes.io/projected/1e74a258-cf66-4f55-b1c9-91e9edac8e40-kube-api-access-6t5xs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.693373 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.693279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:35.972183 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:35.972146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"]
Sep 09 21:19:35.974907 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:19:35.974875    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e74a258_cf66_4f55_b1c9_91e9edac8e40.slice/crio-775a4de26032c3f4028491623a0412019836875913f4f6d96077f2f8a2753e81 WatchSource:0}: Error finding container 775a4de26032c3f4028491623a0412019836875913f4f6d96077f2f8a2753e81: Status 404 returned error can't find the container with id 775a4de26032c3f4028491623a0412019836875913f4f6d96077f2f8a2753e81
Sep 09 21:19:36.509522 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:36.509485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" event={"ID":"1e74a258-cf66-4f55-b1c9-91e9edac8e40","Type":"ContainerStarted","Data":"75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b"}
Sep 09 21:19:36.510017 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:36.509549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" event={"ID":"1e74a258-cf66-4f55-b1c9-91e9edac8e40","Type":"ContainerStarted","Data":"775a4de26032c3f4028491623a0412019836875913f4f6d96077f2f8a2753e81"}
Sep 09 21:19:36.510017 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:36.509579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:19:36.528522 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:19:36.528465    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" podStartSLOduration=1.5284486099999999 podStartE2EDuration="1.52844861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:19:36.528044176 +0000 UTC m=+4101.563515340" watchObservedRunningTime="2026-09-09 21:19:36.52844861 +0000 UTC m=+4101.563919770"
Sep 09 21:20:07.516179 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:07.516067    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:20:20.017123 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.017062    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"]
Sep 09 21:20:20.023195 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.023154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.026240 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.026213    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 09 21:20:20.026411 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.026333    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:20:20.026411 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.026341    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 09 21:20:20.026554 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.026499    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-886t2\""
Sep 09 21:20:20.027783 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.027760    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 09 21:20:20.027889 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.027779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-p4ppj\""
Sep 09 21:20:20.027970 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.027956    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 09 21:20:20.040061 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.040033    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"]
Sep 09 21:20:20.071095 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-model-cache\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.071256 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cn7\" (UniqueName: \"kubernetes.io/projected/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kube-api-access-d7cn7\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.071256 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tls-certs\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.071256 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-home\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.071256 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.071404 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaae88e0-2ff1-41b5-944f-bc316e5d803c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.071404 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.071404 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.071393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-dshm\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172507 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tls-certs\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172688 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-home\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172688 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172688 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaae88e0-2ff1-41b5-944f-bc316e5d803c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172688 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172688 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-dshm\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172962 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-model-cache\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.172962 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cn7\" (UniqueName: \"kubernetes.io/projected/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kube-api-access-d7cn7\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.173078 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.172984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-home\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.173078 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.173027    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.173290 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.173270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.173355 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.173287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-model-cache\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.173633 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.173612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaae88e0-2ff1-41b5-944f-bc316e5d803c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.175254 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.175228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-dshm\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.175452 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.175433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tls-certs\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.182884 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.182859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cn7\" (UniqueName: \"kubernetes.io/projected/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kube-api-access-d7cn7\") pod \"router-with-refs-pd-test-kserve-587dc95d8-g6dj6\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.334205 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.334108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:20.620328 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.620299    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"]
Sep 09 21:20:20.622130 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:20:20.622063    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaae88e0_2ff1_41b5_944f_bc316e5d803c.slice/crio-aa5cdb241728b94f2b1461b042d5a10ad897277b263c036d87c6cd1b67e67317 WatchSource:0}: Error finding container aa5cdb241728b94f2b1461b042d5a10ad897277b263c036d87c6cd1b67e67317: Status 404 returned error can't find the container with id aa5cdb241728b94f2b1461b042d5a10ad897277b263c036d87c6cd1b67e67317
Sep 09 21:20:20.715233 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.715193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerStarted","Data":"bbba44d2d06d8d2934e6df0e5df091a63f5e0f1ce82037800fea83d4cefe977f"}
Sep 09 21:20:20.715408 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.715254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerStarted","Data":"aa5cdb241728b94f2b1461b042d5a10ad897277b263c036d87c6cd1b67e67317"}
Sep 09 21:20:20.715408 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:20.715277    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:21.722034 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:21.721991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerStarted","Data":"a88fb8acf3574da567573f53405ca29b297b09f30d2c4220ff0b1d6fe69c68d1"}
Sep 09 21:20:23.736876 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:23.736837    2566 generic.go:356] "Generic (PLEG): container finished" podID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerID="a88fb8acf3574da567573f53405ca29b297b09f30d2c4220ff0b1d6fe69c68d1" exitCode=0
Sep 09 21:20:23.737338 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:23.736904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerDied","Data":"a88fb8acf3574da567573f53405ca29b297b09f30d2c4220ff0b1d6fe69c68d1"}
Sep 09 21:20:24.745328 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:24.745290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerStarted","Data":"4e3d14a2ebc1de8537b71f997f328890fc36d48ee320d5fca9bee18da88f882d"}
Sep 09 21:20:24.772938 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:24.772864    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podStartSLOduration=3.475837377 podStartE2EDuration="5.772842263s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.297004886s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:24.771422109 +0000 UTC m=+4149.806893269" watchObservedRunningTime="2026-09-09 21:20:24.772842263 +0000 UTC m=+4149.808313424"
Sep 09 21:20:30.334983 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:30.334941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:30.335470 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:30.335007    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:30.336434 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:30.336392    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:20:30.348123 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:30.348067    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:20:40.334989 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:40.334919    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:20:50.335274 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:20:50.335219    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:21:00.335206 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:00.335152    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:21:10.335393 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:10.335329    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:21:20.334690 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:20.334642    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:21:30.335077 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:30.335020    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:21:40.334599 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:40.334486    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:21:43.779351 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:43.779311    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"]
Sep 09 21:21:47.516753 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:47.516704    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:21:50.335076 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:50.335025    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:21:57.515770 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:57.515705    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:21:58.801186 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:21:58.801140    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main" containerID="cri-o://75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b" gracePeriod=45
Sep 09 21:22:00.335001 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:00.334950    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:22:07.516401 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:07.516362    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:07.516855 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:07.516472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:22:10.335496 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:10.335445    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8001/health\": dial tcp 10.134.0.77:8001: connect: connection refused"
Sep 09 21:22:17.516282 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:17.516238    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:20.344031 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:20.343995    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:22:20.356598 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:20.356567    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:22:27.516575 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:27.516531    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:22:29.075294 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.075231    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:22:29.226640 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.226590    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e74a258-cf66-4f55-b1c9-91e9edac8e40-tls-certs\") pod \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") "
Sep 09 21:22:29.226897 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.226715    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t5xs\" (UniqueName: \"kubernetes.io/projected/1e74a258-cf66-4f55-b1c9-91e9edac8e40-kube-api-access-6t5xs\") pod \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\" (UID: \"1e74a258-cf66-4f55-b1c9-91e9edac8e40\") "
Sep 09 21:22:29.230311 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.230261    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e74a258-cf66-4f55-b1c9-91e9edac8e40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e74a258-cf66-4f55-b1c9-91e9edac8e40" (UID: "1e74a258-cf66-4f55-b1c9-91e9edac8e40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:29.230467 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.230345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e74a258-cf66-4f55-b1c9-91e9edac8e40-kube-api-access-6t5xs" (OuterVolumeSpecName: "kube-api-access-6t5xs") pod "1e74a258-cf66-4f55-b1c9-91e9edac8e40" (UID: "1e74a258-cf66-4f55-b1c9-91e9edac8e40"). InnerVolumeSpecName "kube-api-access-6t5xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:29.328773 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.328657    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e74a258-cf66-4f55-b1c9-91e9edac8e40-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:29.328773 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.328701    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t5xs\" (UniqueName: \"kubernetes.io/projected/1e74a258-cf66-4f55-b1c9-91e9edac8e40-kube-api-access-6t5xs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:29.368013 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.367973    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerID="75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b" exitCode=0
Sep 09 21:22:29.368217 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.368015    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"
Sep 09 21:22:29.368217 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.368061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" event={"ID":"1e74a258-cf66-4f55-b1c9-91e9edac8e40","Type":"ContainerDied","Data":"75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b"}
Sep 09 21:22:29.368217 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.368132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246" event={"ID":"1e74a258-cf66-4f55-b1c9-91e9edac8e40","Type":"ContainerDied","Data":"775a4de26032c3f4028491623a0412019836875913f4f6d96077f2f8a2753e81"}
Sep 09 21:22:29.368217 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.368156    2566 scope.go:117] "RemoveContainer" containerID="75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b"
Sep 09 21:22:29.377580 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.377562    2566 scope.go:117] "RemoveContainer" containerID="75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b"
Sep 09 21:22:29.377874 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:22:29.377849    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b\": container with ID starting with 75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b not found: ID does not exist" containerID="75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b"
Sep 09 21:22:29.377968 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.377880    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b"} err="failed to get container status \"75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b\": rpc error: code = NotFound desc = could not find container \"75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b\": container with ID starting with 75d4e019771ccf47f8ae1672c3921d9effbbed19069df03671a93290ffabb71b not found: ID does not exist"
Sep 09 21:22:29.392888 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.392852    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"]
Sep 09 21:22:29.397416 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.397388    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-pj246"]
Sep 09 21:22:29.492358 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.492318    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" path="/var/lib/kubelet/pods/1e74a258-cf66-4f55-b1c9-91e9edac8e40/volumes"
Sep 09 21:22:29.699368 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:29.699329    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"]
Sep 09 21:22:38.245491 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.245458    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"]
Sep 09 21:22:38.245915 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.245830    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main"
Sep 09 21:22:38.245915 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.245841    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main"
Sep 09 21:22:38.245915 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.245897    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e74a258-cf66-4f55-b1c9-91e9edac8e40" containerName="main"
Sep 09 21:22:38.248852 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.248828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.252329 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.252291    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-678q4\""
Sep 09 21:22:38.252329 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.252319    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-wcs7q\""
Sep 09 21:22:38.252538 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.252315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 09 21:22:38.252538 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.252366    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 09 21:22:38.252538 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.252393    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 09 21:22:38.260280 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.260254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"]
Sep 09 21:22:38.417936 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.417898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2cfb89-6612-4625-b823-78a28296719a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.418152 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.417952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qw7z\" (UniqueName: \"kubernetes.io/projected/fc2cfb89-6612-4625-b823-78a28296719a-kube-api-access-8qw7z\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.519178 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.519037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2cfb89-6612-4625-b823-78a28296719a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.519178 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.519129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qw7z\" (UniqueName: \"kubernetes.io/projected/fc2cfb89-6612-4625-b823-78a28296719a-kube-api-access-8qw7z\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.521861 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.521838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2cfb89-6612-4625-b823-78a28296719a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.529272 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.529250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qw7z\" (UniqueName: \"kubernetes.io/projected/fc2cfb89-6612-4625-b823-78a28296719a-kube-api-access-8qw7z\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.561103 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.561046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:38.840841 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:38.840752    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"]
Sep 09 21:22:38.845045 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:22:38.845013    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2cfb89_6612_4625_b823_78a28296719a.slice/crio-a6692f66a35d01aace29f2f83184dad3713dc207aa630a71142966b23644b1e5 WatchSource:0}: Error finding container a6692f66a35d01aace29f2f83184dad3713dc207aa630a71142966b23644b1e5: Status 404 returned error can't find the container with id a6692f66a35d01aace29f2f83184dad3713dc207aa630a71142966b23644b1e5
Sep 09 21:22:39.429029 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:39.428985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" event={"ID":"fc2cfb89-6612-4625-b823-78a28296719a","Type":"ContainerStarted","Data":"f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961"}
Sep 09 21:22:39.429460 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:39.429052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" event={"ID":"fc2cfb89-6612-4625-b823-78a28296719a","Type":"ContainerStarted","Data":"a6692f66a35d01aace29f2f83184dad3713dc207aa630a71142966b23644b1e5"}
Sep 09 21:22:39.429460 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:39.429116    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:22:39.449936 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:39.449881    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" podStartSLOduration=1.449863704 podStartE2EDuration="1.449863704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:39.447564809 +0000 UTC m=+4284.483035963" watchObservedRunningTime="2026-09-09 21:22:39.449863704 +0000 UTC m=+4284.485334864"
Sep 09 21:22:44.723876 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:44.723824    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main" containerID="cri-o://4e3d14a2ebc1de8537b71f997f328890fc36d48ee320d5fca9bee18da88f882d" gracePeriod=45
Sep 09 21:22:45.342677 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.342640    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="llm-d-routing-sidecar" containerID="cri-o://bbba44d2d06d8d2934e6df0e5df091a63f5e0f1ce82037800fea83d4cefe977f" gracePeriod=45
Sep 09 21:22:45.461745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.461712    2566 generic.go:356] "Generic (PLEG): container finished" podID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerID="4e3d14a2ebc1de8537b71f997f328890fc36d48ee320d5fca9bee18da88f882d" exitCode=0
Sep 09 21:22:45.461745 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.461742    2566 generic.go:356] "Generic (PLEG): container finished" podID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerID="bbba44d2d06d8d2934e6df0e5df091a63f5e0f1ce82037800fea83d4cefe977f" exitCode=0
Sep 09 21:22:45.461936 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.461781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerDied","Data":"4e3d14a2ebc1de8537b71f997f328890fc36d48ee320d5fca9bee18da88f882d"}
Sep 09 21:22:45.461936 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.461829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerDied","Data":"bbba44d2d06d8d2934e6df0e5df091a63f5e0f1ce82037800fea83d4cefe977f"}
Sep 09 21:22:45.596264 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.596237    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:22:45.685192 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685146    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-home\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.685379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-dshm\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.685379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7cn7\" (UniqueName: \"kubernetes.io/projected/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kube-api-access-d7cn7\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.685379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685258    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tmp-dir\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.685379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-model-cache\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.685626 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaae88e0-2ff1-41b5-944f-bc316e5d803c-cabundle-cert\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.685711 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685674    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-model-cache" (OuterVolumeSpecName: "model-cache") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.685896 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaae88e0-2ff1-41b5-944f-bc316e5d803c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:45.685969 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.685899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-home" (OuterVolumeSpecName: "home") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.688489 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.688457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-dshm" (OuterVolumeSpecName: "dshm") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.688489 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.688470    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kube-api-access-d7cn7" (OuterVolumeSpecName: "kube-api-access-d7cn7") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "kube-api-access-d7cn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:45.700561 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.700518    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.786441 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786339    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kserve-provision-location\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.786441 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786438    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tls-certs\") pod \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\" (UID: \"eaae88e0-2ff1-41b5-944f-bc316e5d803c\") "
Sep 09 21:22:45.786850 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786644    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.786850 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786659    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaae88e0-2ff1-41b5-944f-bc316e5d803c-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.786850 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786674    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.786850 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786686    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.786850 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786698    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7cn7\" (UniqueName: \"kubernetes.io/projected/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kube-api-access-d7cn7\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.786850 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.786712    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.788731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.788705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:45.809828 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.809779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eaae88e0-2ff1-41b5-944f-bc316e5d803c" (UID: "eaae88e0-2ff1-41b5-944f-bc316e5d803c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:45.887588 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.887552    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaae88e0-2ff1-41b5-944f-bc316e5d803c-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:45.887588 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:45.887584    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaae88e0-2ff1-41b5-944f-bc316e5d803c-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:22:46.468975 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:46.468941    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"
Sep 09 21:22:46.468975 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:46.468964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6" event={"ID":"eaae88e0-2ff1-41b5-944f-bc316e5d803c","Type":"ContainerDied","Data":"aa5cdb241728b94f2b1461b042d5a10ad897277b263c036d87c6cd1b67e67317"}
Sep 09 21:22:46.469240 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:46.469024    2566 scope.go:117] "RemoveContainer" containerID="4e3d14a2ebc1de8537b71f997f328890fc36d48ee320d5fca9bee18da88f882d"
Sep 09 21:22:46.479157 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:46.479135    2566 scope.go:117] "RemoveContainer" containerID="a88fb8acf3574da567573f53405ca29b297b09f30d2c4220ff0b1d6fe69c68d1"
Sep 09 21:22:46.487641 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:46.487621    2566 scope.go:117] "RemoveContainer" containerID="bbba44d2d06d8d2934e6df0e5df091a63f5e0f1ce82037800fea83d4cefe977f"
Sep 09 21:22:46.495609 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:46.495580    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"]
Sep 09 21:22:46.499624 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:46.499598    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-587dc95d8-g6dj6"]
Sep 09 21:22:47.492791 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:22:47.492745    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" path="/var/lib/kubelet/pods/eaae88e0-2ff1-41b5-944f-bc316e5d803c/volumes"
Sep 09 21:23:10.436469 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:23:10.436375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:24:03.285668 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.285627    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"]
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.285982    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main"
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.285994    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main"
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.286009    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="llm-d-routing-sidecar"
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.286015    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="llm-d-routing-sidecar"
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.286033    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="storage-initializer"
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.286039    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="storage-initializer"
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.286104    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="llm-d-routing-sidecar"
Sep 09 21:24:03.286119 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.286116    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaae88e0-2ff1-41b5-944f-bc316e5d803c" containerName="main"
Sep 09 21:24:03.289225 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.289206    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.298918 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.298885    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"]
Sep 09 21:24:03.305661 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.305637    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jxfn7\""
Sep 09 21:24:03.305802 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.305638    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 09 21:24:03.305802 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.305638    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 09 21:24:03.306651 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.306635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 09 21:24:03.341984 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.341948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.341984 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.341985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.342213 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.342004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.342213 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.342061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcn7\" (UniqueName: \"kubernetes.io/projected/ba11b302-64cb-4c64-8d8c-29c6f7afe412-kube-api-access-chcn7\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.342213 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.342159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.342213 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.342195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443151 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443335 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443335 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcn7\" (UniqueName: \"kubernetes.io/projected/ba11b302-64cb-4c64-8d8c-29c6f7afe412-kube-api-access-chcn7\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443335 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443335 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443563 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443610 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443662 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.443731 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.443717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.445693 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.445672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.446046 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.446028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.452438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.452414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcn7\" (UniqueName: \"kubernetes.io/projected/ba11b302-64cb-4c64-8d8c-29c6f7afe412-kube-api-access-chcn7\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-vd87b\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.599749 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.599657    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:03.874064 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:03.874034    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"]
Sep 09 21:24:03.876242 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:24:03.876216    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba11b302_64cb_4c64_8d8c_29c6f7afe412.slice/crio-614bf45702a00cdbdf6b7bbe163201e52a1d2a997867e41d6a2b9f440c13a071 WatchSource:0}: Error finding container 614bf45702a00cdbdf6b7bbe163201e52a1d2a997867e41d6a2b9f440c13a071: Status 404 returned error can't find the container with id 614bf45702a00cdbdf6b7bbe163201e52a1d2a997867e41d6a2b9f440c13a071
Sep 09 21:24:04.839452 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:04.839415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" event={"ID":"ba11b302-64cb-4c64-8d8c-29c6f7afe412","Type":"ContainerStarted","Data":"37389479dde7d25b2ba6a1f70caa60d37007cac77905bc5e030f63b0fa8ff710"}
Sep 09 21:24:04.839911 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:04.839471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" event={"ID":"ba11b302-64cb-4c64-8d8c-29c6f7afe412","Type":"ContainerStarted","Data":"614bf45702a00cdbdf6b7bbe163201e52a1d2a997867e41d6a2b9f440c13a071"}
Sep 09 21:24:04.864442 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:04.864380    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" podStartSLOduration=1.8643617959999998 podStartE2EDuration="1.864361796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:24:04.861955331 +0000 UTC m=+4369.897426491" watchObservedRunningTime="2026-09-09 21:24:04.864361796 +0000 UTC m=+4369.899832955"
Sep 09 21:24:05.600348 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:05.600310    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:13.600562 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:13.600515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:13.613409 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:13.613378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:13.897249 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:13.897216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:21.599147 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:21.599109    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"]
Sep 09 21:24:21.608978 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:24:21.608938    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:24:21.608978 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:21.608978 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:21.608978 ip-10-0-131-185 kubenswrapper[2566]:  > containerID="37389479dde7d25b2ba6a1f70caa60d37007cac77905bc5e030f63b0fa8ff710" cmd=["/bin/sleep","15"]
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:21.608999    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" message="" err=<
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]:  >
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:24:21.609018    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" podUID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" containerName="main" containerID="cri-o://37389479dde7d25b2ba6a1f70caa60d37007cac77905bc5e030f63b0fa8ff710"
Sep 09 21:24:21.609349 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:21.609059    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" podUID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" containerName="main" containerID="cri-o://37389479dde7d25b2ba6a1f70caa60d37007cac77905bc5e030f63b0fa8ff710" gracePeriod=60
Sep 09 21:24:21.927286 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:21.927251    2566 generic.go:356] "Generic (PLEG): container finished" podID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" containerID="37389479dde7d25b2ba6a1f70caa60d37007cac77905bc5e030f63b0fa8ff710" exitCode=0
Sep 09 21:24:21.927489 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:21.927328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" event={"ID":"ba11b302-64cb-4c64-8d8c-29c6f7afe412","Type":"ContainerDied","Data":"37389479dde7d25b2ba6a1f70caa60d37007cac77905bc5e030f63b0fa8ff710"}
Sep 09 21:24:21.951910 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:21.951881    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:22.109041 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.108995    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chcn7\" (UniqueName: \"kubernetes.io/projected/ba11b302-64cb-4c64-8d8c-29c6f7afe412-kube-api-access-chcn7\") pod \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") "
Sep 09 21:24:22.109041 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109044    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tls-certs\") pod \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") "
Sep 09 21:24:22.109331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tmp-dir\") pod \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") "
Sep 09 21:24:22.109331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-model-cache\") pod \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") "
Sep 09 21:24:22.109331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109212    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-home\") pod \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") "
Sep 09 21:24:22.109331 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-dshm\") pod \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\" (UID: \"ba11b302-64cb-4c64-8d8c-29c6f7afe412\") "
Sep 09 21:24:22.109581 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-home" (OuterVolumeSpecName: "home") pod "ba11b302-64cb-4c64-8d8c-29c6f7afe412" (UID: "ba11b302-64cb-4c64-8d8c-29c6f7afe412"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:22.109581 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba11b302-64cb-4c64-8d8c-29c6f7afe412" (UID: "ba11b302-64cb-4c64-8d8c-29c6f7afe412"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:22.109581 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109521    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba11b302-64cb-4c64-8d8c-29c6f7afe412" (UID: "ba11b302-64cb-4c64-8d8c-29c6f7afe412"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:22.109692 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109627    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:24:22.109692 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109642    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:24:22.109692 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.109650    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:24:22.111595 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.111573    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-dshm" (OuterVolumeSpecName: "dshm") pod "ba11b302-64cb-4c64-8d8c-29c6f7afe412" (UID: "ba11b302-64cb-4c64-8d8c-29c6f7afe412"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:24:22.111919 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.111896    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba11b302-64cb-4c64-8d8c-29c6f7afe412" (UID: "ba11b302-64cb-4c64-8d8c-29c6f7afe412"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:24:22.111919 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.111911    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba11b302-64cb-4c64-8d8c-29c6f7afe412-kube-api-access-chcn7" (OuterVolumeSpecName: "kube-api-access-chcn7") pod "ba11b302-64cb-4c64-8d8c-29c6f7afe412" (UID: "ba11b302-64cb-4c64-8d8c-29c6f7afe412"). InnerVolumeSpecName "kube-api-access-chcn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:24:22.210276 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.210184    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba11b302-64cb-4c64-8d8c-29c6f7afe412-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:24:22.210276 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.210217    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcn7\" (UniqueName: \"kubernetes.io/projected/ba11b302-64cb-4c64-8d8c-29c6f7afe412-kube-api-access-chcn7\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:24:22.210276 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.210229    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba11b302-64cb-4c64-8d8c-29c6f7afe412-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:24:22.883453 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.883403    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" podUID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:24:22.933154 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.933122    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"
Sep 09 21:24:22.933336 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.933133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b" event={"ID":"ba11b302-64cb-4c64-8d8c-29c6f7afe412","Type":"ContainerDied","Data":"614bf45702a00cdbdf6b7bbe163201e52a1d2a997867e41d6a2b9f440c13a071"}
Sep 09 21:24:22.933336 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.933256    2566 scope.go:117] "RemoveContainer" containerID="37389479dde7d25b2ba6a1f70caa60d37007cac77905bc5e030f63b0fa8ff710"
Sep 09 21:24:22.963373 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.963347    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"]
Sep 09 21:24:22.969958 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:22.969935    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-vd87b"]
Sep 09 21:24:23.491436 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:23.491395    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" path="/var/lib/kubelet/pods/ba11b302-64cb-4c64-8d8c-29c6f7afe412/volumes"
Sep 09 21:24:39.744153 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.744049    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"]
Sep 09 21:24:39.744723 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.744608    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" containerName="main"
Sep 09 21:24:39.744723 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.744630    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" containerName="main"
Sep 09 21:24:39.744723 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.744718    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba11b302-64cb-4c64-8d8c-29c6f7afe412" containerName="main"
Sep 09 21:24:39.747921 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.747898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.750656 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.750623    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 09 21:24:39.750782 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.750623    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:24:39.755986 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.755959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tls-certs\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.756129 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.755996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-home\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.756129 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.756017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tmp-dir\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.756262 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.756168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbv9\" (UniqueName: \"kubernetes.io/projected/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kube-api-access-xbbv9\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.756262 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.756210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-cabundle-cert\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.756383 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.756274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kserve-provision-location\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.756383 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.756324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-model-cache\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.756499 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.756351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-dshm\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.761799 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.761775    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"]
Sep 09 21:24:39.857191 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbv9\" (UniqueName: \"kubernetes.io/projected/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kube-api-access-xbbv9\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-cabundle-cert\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kserve-provision-location\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-model-cache\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-dshm\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857379 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tls-certs\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857643 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-home\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857643 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tmp-dir\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857751 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857719    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kserve-provision-location\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.857812 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-model-cache\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.858030 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.857999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-home\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.858199 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.858058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tmp-dir\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.858199 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.858166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-cabundle-cert\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.860025 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.860003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-dshm\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.860291 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.860271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tls-certs\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:39.868392 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:39.868369    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbv9\" (UniqueName: \"kubernetes.io/projected/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kube-api-access-xbbv9\") pod \"prestop-hook-test-kserve-78bfbf4ccd-m7g77\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:40.058097 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:40.057987    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:40.339856 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:40.339824    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"]
Sep 09 21:24:40.342183 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:24:40.342146    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4cec9f_19dd_4de8_855e_eba9f5aa237a.slice/crio-226c32c8c9e2eac9cd7fee3d43beebc8545883da498e594f978adfb2fd82ea8a WatchSource:0}: Error finding container 226c32c8c9e2eac9cd7fee3d43beebc8545883da498e594f978adfb2fd82ea8a: Status 404 returned error can't find the container with id 226c32c8c9e2eac9cd7fee3d43beebc8545883da498e594f978adfb2fd82ea8a
Sep 09 21:24:41.020422 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:41.020372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" event={"ID":"ef4cec9f-19dd-4de8-855e-eba9f5aa237a","Type":"ContainerStarted","Data":"6d6620f7f9c9cbbf392a5c000680fe14ea0437f9a609c0005c86aabbec5b25a9"}
Sep 09 21:24:41.020810 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:41.020479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" event={"ID":"ef4cec9f-19dd-4de8-855e-eba9f5aa237a","Type":"ContainerStarted","Data":"226c32c8c9e2eac9cd7fee3d43beebc8545883da498e594f978adfb2fd82ea8a"}
Sep 09 21:24:43.034860 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:43.034770    2566 generic.go:356] "Generic (PLEG): container finished" podID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerID="6d6620f7f9c9cbbf392a5c000680fe14ea0437f9a609c0005c86aabbec5b25a9" exitCode=0
Sep 09 21:24:43.035359 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:43.034847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" event={"ID":"ef4cec9f-19dd-4de8-855e-eba9f5aa237a","Type":"ContainerDied","Data":"6d6620f7f9c9cbbf392a5c000680fe14ea0437f9a609c0005c86aabbec5b25a9"}
Sep 09 21:24:44.043310 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:44.043272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" event={"ID":"ef4cec9f-19dd-4de8-855e-eba9f5aa237a","Type":"ContainerStarted","Data":"c00d9507b7882df783f399a95bf80f8f010b54c7176ea6b2a3a99646c0c58a26"}
Sep 09 21:24:44.058503 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:44.058469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:44.070364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:44.070306    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" podStartSLOduration=2.797401243 podStartE2EDuration="5.070290305s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.272889062s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:24:44.067244555 +0000 UTC m=+4409.102715736" watchObservedRunningTime="2026-09-09 21:24:44.070290305 +0000 UTC m=+4409.105761499"
Sep 09 21:24:50.058269 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:50.058222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:24:50.060241 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:24:50.060208    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 09 21:25:00.058877 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:00.058825    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 09 21:25:06.350362 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:06.350320    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"]
Sep 09 21:25:06.360561 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:06.360526    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"]
Sep 09 21:25:10.436803 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:10.436760    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:25:20.436336 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:20.436292    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:25:21.374883 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:21.374806    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="main" containerID="cri-o://c00d9507b7882df783f399a95bf80f8f010b54c7176ea6b2a3a99646c0c58a26" gracePeriod=45
Sep 09 21:25:21.380683 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:21.380629    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main" containerID="cri-o://f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961" gracePeriod=45
Sep 09 21:25:23.241610 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.241583    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-78bfbf4ccd-m7g77_ef4cec9f-19dd-4de8-855e-eba9f5aa237a/main/0.log"
Sep 09 21:25:23.242100 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.242057    2566 generic.go:356] "Generic (PLEG): container finished" podID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerID="c00d9507b7882df783f399a95bf80f8f010b54c7176ea6b2a3a99646c0c58a26" exitCode=130
Sep 09 21:25:23.242182 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.242115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" event={"ID":"ef4cec9f-19dd-4de8-855e-eba9f5aa237a","Type":"ContainerDied","Data":"c00d9507b7882df783f399a95bf80f8f010b54c7176ea6b2a3a99646c0c58a26"}
Sep 09 21:25:23.318190 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.318159    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-78bfbf4ccd-m7g77_ef4cec9f-19dd-4de8-855e-eba9f5aa237a/main/0.log"
Sep 09 21:25:23.320850 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.320829    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:25:23.347281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347249    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tmp-dir\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347281 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347288    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-dshm\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347530 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tls-certs\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347530 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347402    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbbv9\" (UniqueName: \"kubernetes.io/projected/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kube-api-access-xbbv9\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347661 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347566    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-home\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347661 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347626    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-cabundle-cert\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347661 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347625    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:23.347825 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347666    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-model-cache\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347825 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kserve-provision-location\") pod \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\" (UID: \"ef4cec9f-19dd-4de8-855e-eba9f5aa237a\") "
Sep 09 21:25:23.347825 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.347810    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-home" (OuterVolumeSpecName: "home") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:23.348766 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.348723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:23.349417 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.349147    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-home\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:23.349417 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.349174    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-model-cache\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:23.349417 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.349230    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tmp-dir\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:23.349635 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.349475    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:25:23.360295 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.350222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:25:23.360295 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.351387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kube-api-access-xbbv9" (OuterVolumeSpecName: "kube-api-access-xbbv9") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "kube-api-access-xbbv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:25:23.360295 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.351934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-dshm" (OuterVolumeSpecName: "dshm") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:23.382966 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.382866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef4cec9f-19dd-4de8-855e-eba9f5aa237a" (UID: "ef4cec9f-19dd-4de8-855e-eba9f5aa237a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:23.449680 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.449636    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kserve-provision-location\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:23.449680 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.449673    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-dshm\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:23.449680 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.449685    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:23.449927 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.449696    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbbv9\" (UniqueName: \"kubernetes.io/projected/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-kube-api-access-xbbv9\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:23.449927 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:23.449704    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef4cec9f-19dd-4de8-855e-eba9f5aa237a-cabundle-cert\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:24.249588 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:24.249552    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-78bfbf4ccd-m7g77_ef4cec9f-19dd-4de8-855e-eba9f5aa237a/main/0.log"
Sep 09 21:25:24.250041 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:24.250018    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"
Sep 09 21:25:24.250041 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:24.250025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77" event={"ID":"ef4cec9f-19dd-4de8-855e-eba9f5aa237a","Type":"ContainerDied","Data":"226c32c8c9e2eac9cd7fee3d43beebc8545883da498e594f978adfb2fd82ea8a"}
Sep 09 21:25:24.250220 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:24.250118    2566 scope.go:117] "RemoveContainer" containerID="c00d9507b7882df783f399a95bf80f8f010b54c7176ea6b2a3a99646c0c58a26"
Sep 09 21:25:24.259438 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:24.259413    2566 scope.go:117] "RemoveContainer" containerID="6d6620f7f9c9cbbf392a5c000680fe14ea0437f9a609c0005c86aabbec5b25a9"
Sep 09 21:25:24.271454 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:24.271422    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"]
Sep 09 21:25:24.276815 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:24.276781    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-78bfbf4ccd-m7g77"]
Sep 09 21:25:25.491784 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:25.491749    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" path="/var/lib/kubelet/pods/ef4cec9f-19dd-4de8-855e-eba9f5aa237a/volumes"
Sep 09 21:25:30.436557 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:30.436517    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:25:30.437056 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:30.436615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:25:40.435736 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:40.435698    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:25:50.436007 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:50.435961    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:25:51.641209 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:51.641181    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:25:51.796417 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:51.796313    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2cfb89-6612-4625-b823-78a28296719a-tls-certs\") pod \"fc2cfb89-6612-4625-b823-78a28296719a\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") "
Sep 09 21:25:51.796417 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:51.796401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qw7z\" (UniqueName: \"kubernetes.io/projected/fc2cfb89-6612-4625-b823-78a28296719a-kube-api-access-8qw7z\") pod \"fc2cfb89-6612-4625-b823-78a28296719a\" (UID: \"fc2cfb89-6612-4625-b823-78a28296719a\") "
Sep 09 21:25:51.798567 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:51.798532    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2cfb89-6612-4625-b823-78a28296719a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc2cfb89-6612-4625-b823-78a28296719a" (UID: "fc2cfb89-6612-4625-b823-78a28296719a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:25:51.798694 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:51.798628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2cfb89-6612-4625-b823-78a28296719a-kube-api-access-8qw7z" (OuterVolumeSpecName: "kube-api-access-8qw7z") pod "fc2cfb89-6612-4625-b823-78a28296719a" (UID: "fc2cfb89-6612-4625-b823-78a28296719a"). InnerVolumeSpecName "kube-api-access-8qw7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:25:51.897823 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:51.897774    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qw7z\" (UniqueName: \"kubernetes.io/projected/fc2cfb89-6612-4625-b823-78a28296719a-kube-api-access-8qw7z\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:51.897823 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:51.897817    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2cfb89-6612-4625-b823-78a28296719a-tls-certs\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:25:52.379736 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.379699    2566 generic.go:356] "Generic (PLEG): container finished" podID="fc2cfb89-6612-4625-b823-78a28296719a" containerID="f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961" exitCode=0
Sep 09 21:25:52.379926 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.379744    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"
Sep 09 21:25:52.379926 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.379785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" event={"ID":"fc2cfb89-6612-4625-b823-78a28296719a","Type":"ContainerDied","Data":"f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961"}
Sep 09 21:25:52.379926 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.379834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl" event={"ID":"fc2cfb89-6612-4625-b823-78a28296719a","Type":"ContainerDied","Data":"a6692f66a35d01aace29f2f83184dad3713dc207aa630a71142966b23644b1e5"}
Sep 09 21:25:52.379926 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.379865    2566 scope.go:117] "RemoveContainer" containerID="f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961"
Sep 09 21:25:52.389282 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.389260    2566 scope.go:117] "RemoveContainer" containerID="f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961"
Sep 09 21:25:52.389548 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:25:52.389526    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961\": container with ID starting with f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961 not found: ID does not exist" containerID="f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961"
Sep 09 21:25:52.389626 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.389558    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961"} err="failed to get container status \"f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961\": rpc error: code = NotFound desc = could not find container \"f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961\": container with ID starting with f5d862d258f1000e0c264293ea330b289b2fff427f411318c4af7f330392c961 not found: ID does not exist"
Sep 09 21:25:52.403880 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.403851    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"]
Sep 09 21:25:52.409109 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:52.409068    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-g47rl"]
Sep 09 21:25:53.491932 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:53.491897    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2cfb89-6612-4625-b823-78a28296719a" path="/var/lib/kubelet/pods/fc2cfb89-6612-4625-b823-78a28296719a/volumes"
Sep 09 21:25:56.480364 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480325    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4x48b/must-gather-jb5vm"]
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480668    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="storage-initializer"
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480680    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="storage-initializer"
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480691    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main"
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480696    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main"
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480705    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="main"
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480711    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="main"
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480771    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef4cec9f-19dd-4de8-855e-eba9f5aa237a" containerName="main"
Sep 09 21:25:56.480830 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.480779    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2cfb89-6612-4625-b823-78a28296719a" containerName="main"
Sep 09 21:25:56.485401 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.485379    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:56.487844 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.487821    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4x48b\"/\"kube-root-ca.crt\""
Sep 09 21:25:56.488160 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.488142    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4x48b\"/\"openshift-service-ca.crt\""
Sep 09 21:25:56.495286 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.495261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4x48b/must-gather-jb5vm"]
Sep 09 21:25:56.642582 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.642542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bth\" (UniqueName: \"kubernetes.io/projected/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-kube-api-access-t5bth\") pod \"must-gather-jb5vm\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") " pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:56.642764 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.642613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-must-gather-output\") pod \"must-gather-jb5vm\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") " pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:56.743609 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.743499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-must-gather-output\") pod \"must-gather-jb5vm\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") " pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:56.743794 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.743616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bth\" (UniqueName: \"kubernetes.io/projected/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-kube-api-access-t5bth\") pod \"must-gather-jb5vm\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") " pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:56.743887 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.743865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-must-gather-output\") pod \"must-gather-jb5vm\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") " pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:56.753136 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.753101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bth\" (UniqueName: \"kubernetes.io/projected/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-kube-api-access-t5bth\") pod \"must-gather-jb5vm\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") " pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:56.795172 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:56.795131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:25:57.067485 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:57.067458    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4x48b/must-gather-jb5vm"]
Sep 09 21:25:57.069790 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:25:57.069756    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a4b74a_ec2c_4157_8bf5_e371d14c0bc3.slice/crio-c9baad2257df71a63781229cf026d444660a079a41325edc0adf825a1b8b53f0 WatchSource:0}: Error finding container c9baad2257df71a63781229cf026d444660a079a41325edc0adf825a1b8b53f0: Status 404 returned error can't find the container with id c9baad2257df71a63781229cf026d444660a079a41325edc0adf825a1b8b53f0
Sep 09 21:25:57.071578 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:57.071558    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:25:57.404111 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:25:57.404054    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4x48b/must-gather-jb5vm" event={"ID":"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3","Type":"ContainerStarted","Data":"c9baad2257df71a63781229cf026d444660a079a41325edc0adf825a1b8b53f0"}
Sep 09 21:26:03.448248 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:03.448175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4x48b/must-gather-jb5vm" event={"ID":"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3","Type":"ContainerStarted","Data":"4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01"}
Sep 09 21:26:04.455658 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:04.455600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4x48b/must-gather-jb5vm" event={"ID":"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3","Type":"ContainerStarted","Data":"e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390"}
Sep 09 21:26:04.475574 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:04.475519    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4x48b/must-gather-jb5vm" podStartSLOduration=2.260484207 podStartE2EDuration="8.475501857s" totalImagesPullingTime="6.21501765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:04.473609033 +0000 UTC m=+4489.509080193" watchObservedRunningTime="2026-09-09 21:26:04.475501857 +0000 UTC m=+4489.510973016"
Sep 09 21:26:19.976789 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:19.976758    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d7ff7ffd8-b545h_681323ae-d7bd-43ce-8bc6-34b06c1df3b8/router/0.log"
Sep 09 21:26:20.809930 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:20.809904    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d7ff7ffd8-b545h_681323ae-d7bd-43ce-8bc6-34b06c1df3b8/router/0.log"
Sep 09 21:26:21.989189 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:21.989143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-z6mfr_63b8a9e1-d5c9-4ec9-b774-ae9607616c88/authorino/0.log"
Sep 09 21:26:22.055636 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:22.055605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kr2db_16a612c1-9152-41a3-952a-5e85f2de5bb1/kuadrant-console-plugin/0.log"
Sep 09 21:26:27.580682 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:27.580643    2566 generic.go:356] "Generic (PLEG): container finished" podID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerID="4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01" exitCode=0
Sep 09 21:26:27.581183 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:27.580718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4x48b/must-gather-jb5vm" event={"ID":"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3","Type":"ContainerDied","Data":"4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01"}
Sep 09 21:26:27.581183 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:27.581068    2566 scope.go:117] "RemoveContainer" containerID="4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01"
Sep 09 21:26:27.649767 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:27.649714    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4x48b_must-gather-jb5vm_f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3/gather/0.log"
Sep 09 21:26:31.344355 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:31.344324    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tq7t4_78e54e15-216a-4f30-ac4e-b1848d33090b/global-pull-secret-syncer/0.log"
Sep 09 21:26:31.511297 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:31.511266    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tvmd6_7365b6cd-5c52-4083-aeec-55909a783f40/konnectivity-agent/0.log"
Sep 09 21:26:31.536486 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:31.536452    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-185.ec2.internal_7b0eae04d84f726f4b8e2b038df85374/haproxy/0.log"
Sep 09 21:26:33.094692 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.094645    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-4x48b/must-gather-jb5vm"]
Sep 09 21:26:33.095327 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.094899    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-4x48b/must-gather-jb5vm" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerName="copy" containerID="cri-o://e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390" gracePeriod=2
Sep 09 21:26:33.097129 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:26:33.097067    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jb5vm\" is forbidden: User \"system:node:ip-10-0-131-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4x48b\": no relationship found between node 'ip-10-0-131-185.ec2.internal' and this object" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:26:33.099619 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.098990    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-4x48b/must-gather-jb5vm"]
Sep 09 21:26:33.331781 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.331758    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4x48b_must-gather-jb5vm_f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3/copy/0.log"
Sep 09 21:26:33.334525 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.334506    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:26:33.337555 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:26:33.337526    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jb5vm\" is forbidden: User \"system:node:ip-10-0-131-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-4x48b\": no relationship found between node 'ip-10-0-131-185.ec2.internal' and this object" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:26:33.470265 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.470224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-must-gather-output\") pod \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") "
Sep 09 21:26:33.470476 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.470291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5bth\" (UniqueName: \"kubernetes.io/projected/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-kube-api-access-t5bth\") pod \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\" (UID: \"f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3\") "
Sep 09 21:26:33.472768 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.472734    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-kube-api-access-t5bth" (OuterVolumeSpecName: "kube-api-access-t5bth") pod "f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" (UID: "f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3"). InnerVolumeSpecName "kube-api-access-t5bth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:26:33.476984 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.476957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" (UID: "f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:26:33.492038 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.492008    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" path="/var/lib/kubelet/pods/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3/volumes"
Sep 09 21:26:33.571992 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.571945    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-must-gather-output\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:26:33.571992 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.571988    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5bth\" (UniqueName: \"kubernetes.io/projected/f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3-kube-api-access-t5bth\") on node \"ip-10-0-131-185.ec2.internal\" DevicePath \"\""
Sep 09 21:26:33.610933 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.610906    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-4x48b_must-gather-jb5vm_f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3/copy/0.log"
Sep 09 21:26:33.611300 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.611278    2566 generic.go:356] "Generic (PLEG): container finished" podID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerID="e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390" exitCode=143
Sep 09 21:26:33.611381 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.611329    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4x48b/must-gather-jb5vm"
Sep 09 21:26:33.611381 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.611370    2566 scope.go:117] "RemoveContainer" containerID="e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390"
Sep 09 21:26:33.619643 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.619620    2566 scope.go:117] "RemoveContainer" containerID="4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01"
Sep 09 21:26:33.632427 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.632407    2566 scope.go:117] "RemoveContainer" containerID="e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390"
Sep 09 21:26:33.632758 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:26:33.632729    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390\": container with ID starting with e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390 not found: ID does not exist" containerID="e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390"
Sep 09 21:26:33.632901 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.632764    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390"} err="failed to get container status \"e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390\": rpc error: code = NotFound desc = could not find container \"e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390\": container with ID starting with e3a7b7e848d7dfbfda89c292a39b57bbd89d21b09b08f5109b3cd7b0cab23390 not found: ID does not exist"
Sep 09 21:26:33.632901 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.632784    2566 scope.go:117] "RemoveContainer" containerID="4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01"
Sep 09 21:26:33.633103 ip-10-0-131-185 kubenswrapper[2566]: E0909 21:26:33.633058    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01\": container with ID starting with 4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01 not found: ID does not exist" containerID="4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01"
Sep 09 21:26:33.633172 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:33.633102    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01"} err="failed to get container status \"4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01\": rpc error: code = NotFound desc = could not find container \"4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01\": container with ID starting with 4e60b37dc95ec01e7fe47d4dfd7c7956e2dd5a8e5c02da92cb87fcd39efa9e01 not found: ID does not exist"
Sep 09 21:26:36.650396 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:36.648299    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-z6mfr_63b8a9e1-d5c9-4ec9-b774-ae9607616c88/authorino/0.log"
Sep 09 21:26:36.751189 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:36.751155    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kr2db_16a612c1-9152-41a3-952a-5e85f2de5bb1/kuadrant-console-plugin/0.log"
Sep 09 21:26:38.125418 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.125390    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d9a68be8-3474-4e0d-bb91-2d3b2b635122/alertmanager/0.log"
Sep 09 21:26:38.173202 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.173175    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d9a68be8-3474-4e0d-bb91-2d3b2b635122/config-reloader/0.log"
Sep 09 21:26:38.216217 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.216183    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d9a68be8-3474-4e0d-bb91-2d3b2b635122/kube-rbac-proxy-web/0.log"
Sep 09 21:26:38.256522 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.256484    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d9a68be8-3474-4e0d-bb91-2d3b2b635122/kube-rbac-proxy/0.log"
Sep 09 21:26:38.281221 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.281196    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d9a68be8-3474-4e0d-bb91-2d3b2b635122/kube-rbac-proxy-metric/0.log"
Sep 09 21:26:38.307822 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.307798    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d9a68be8-3474-4e0d-bb91-2d3b2b635122/prom-label-proxy/0.log"
Sep 09 21:26:38.335040 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.335014    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d9a68be8-3474-4e0d-bb91-2d3b2b635122/init-config-reloader/0.log"
Sep 09 21:26:38.421122 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.421026    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6wspw_ef352ed3-6351-47a1-98f6-998e9902b2fa/kube-state-metrics/0.log"
Sep 09 21:26:38.447356 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.447324    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6wspw_ef352ed3-6351-47a1-98f6-998e9902b2fa/kube-rbac-proxy-main/0.log"
Sep 09 21:26:38.477618 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.477565    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6wspw_ef352ed3-6351-47a1-98f6-998e9902b2fa/kube-rbac-proxy-self/0.log"
Sep 09 21:26:38.691661 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.691585    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbs5f_9b0a0be3-99f1-41a9-8119-927574ac4f68/node-exporter/0.log"
Sep 09 21:26:38.727214 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.727187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbs5f_9b0a0be3-99f1-41a9-8119-927574ac4f68/kube-rbac-proxy/0.log"
Sep 09 21:26:38.753814 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.753782    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bbs5f_9b0a0be3-99f1-41a9-8119-927574ac4f68/init-textfile/0.log"
Sep 09 21:26:38.875940 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.875907    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wx6pv_0480caf2-ce19-4d3a-830d-30a27f5854ef/kube-rbac-proxy-main/0.log"
Sep 09 21:26:38.906243 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.906210    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wx6pv_0480caf2-ce19-4d3a-830d-30a27f5854ef/kube-rbac-proxy-self/0.log"
Sep 09 21:26:38.932989 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:38.932944    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wx6pv_0480caf2-ce19-4d3a-830d-30a27f5854ef/openshift-state-metrics/0.log"
Sep 09 21:26:39.280206 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:39.280176    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d88bb879-jvq2g_535c62cd-c525-45c8-894c-159be2a18b7d/telemeter-client/0.log"
Sep 09 21:26:39.311177 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:39.311147    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d88bb879-jvq2g_535c62cd-c525-45c8-894c-159be2a18b7d/reload/0.log"
Sep 09 21:26:39.338662 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:39.338632    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d88bb879-jvq2g_535c62cd-c525-45c8-894c-159be2a18b7d/kube-rbac-proxy/0.log"
Sep 09 21:26:41.165941 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.165905    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"]
Sep 09 21:26:41.166497 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.166462    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerName="copy"
Sep 09 21:26:41.166497 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.166486    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerName="copy"
Sep 09 21:26:41.166622 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.166507    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerName="gather"
Sep 09 21:26:41.166622 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.166514    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerName="gather"
Sep 09 21:26:41.166622 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.166584    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerName="copy"
Sep 09 21:26:41.166622 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.166595    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a4b74a-ec2c-4157-8bf5-e371d14c0bc3" containerName="gather"
Sep 09 21:26:41.169486 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.169464    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.172536 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.172517    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sghw5\"/\"kube-root-ca.crt\""
Sep 09 21:26:41.172646 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.172517    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sghw5\"/\"default-dockercfg-mcj2p\""
Sep 09 21:26:41.173175 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.173155    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sghw5\"/\"openshift-service-ca.crt\""
Sep 09 21:26:41.180902 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.180876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"]
Sep 09 21:26:41.238579 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.238539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-podres\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.238788 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.238601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-proc\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.238788 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.238668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-sys\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.238788 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.238733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-lib-modules\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.238950 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.238830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7wd\" (UniqueName: \"kubernetes.io/projected/7f0c12d0-fb81-46e3-bee2-729032df490a-kube-api-access-jq7wd\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340027 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.339984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7wd\" (UniqueName: \"kubernetes.io/projected/7f0c12d0-fb81-46e3-bee2-729032df490a-kube-api-access-jq7wd\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340027 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-podres\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340333 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-proc\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340333 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-sys\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340333 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-lib-modules\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340333 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-proc\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340333 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-podres\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340333 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-sys\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.340333 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.340271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f0c12d0-fb81-46e3-bee2-729032df490a-lib-modules\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.349671 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.349648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7wd\" (UniqueName: \"kubernetes.io/projected/7f0c12d0-fb81-46e3-bee2-729032df490a-kube-api-access-jq7wd\") pod \"perf-node-gather-daemonset-r4kt8\" (UID: \"7f0c12d0-fb81-46e3-bee2-729032df490a\") " pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.480507 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.480425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:41.752439 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.752362    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"]
Sep 09 21:26:41.755795 ip-10-0-131-185 kubenswrapper[2566]: W0909 21:26:41.755766    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f0c12d0_fb81_46e3_bee2_729032df490a.slice/crio-81fc3e0b862163c2b84615ed18a5f456fb65f2916fce124893a7de45e44cf273 WatchSource:0}: Error finding container 81fc3e0b862163c2b84615ed18a5f456fb65f2916fce124893a7de45e44cf273: Status 404 returned error can't find the container with id 81fc3e0b862163c2b84615ed18a5f456fb65f2916fce124893a7de45e44cf273
Sep 09 21:26:41.802098 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:41.802058    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-r92mr_823113ee-0dba-419e-bb10-80c0eb4f0a92/download-server/0.log"
Sep 09 21:26:42.656982 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:42.656943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8" event={"ID":"7f0c12d0-fb81-46e3-bee2-729032df490a","Type":"ContainerStarted","Data":"ff2eec2c357af99efb9b9996c14866bcb3fbf7f50bc0b11ec4f8e70707708ed5"}
Sep 09 21:26:42.657390 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:42.657012    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:42.657390 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:42.657033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8" event={"ID":"7f0c12d0-fb81-46e3-bee2-729032df490a","Type":"ContainerStarted","Data":"81fc3e0b862163c2b84615ed18a5f456fb65f2916fce124893a7de45e44cf273"}
Sep 09 21:26:42.676499 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:42.676426    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8" podStartSLOduration=1.676412536 podStartE2EDuration="1.676412536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:42.674657682 +0000 UTC m=+4527.710128847" watchObservedRunningTime="2026-09-09 21:26:42.676412536 +0000 UTC m=+4527.711883696"
Sep 09 21:26:43.511685 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:43.511658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jthx2_1e40077b-94d8-4b82-8982-969b50b1fe96/dns/0.log"
Sep 09 21:26:43.535904 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:43.535878    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jthx2_1e40077b-94d8-4b82-8982-969b50b1fe96/kube-rbac-proxy/0.log"
Sep 09 21:26:43.672326 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:43.672297    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pt8r9_5a62488d-cc14-4591-a50f-1665c9ad2800/dns-node-resolver/0.log"
Sep 09 21:26:44.227853 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:44.227819    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-869956756f-v882x_5fb96632-7127-4e2a-b7af-285de927da6d/registry/0.log"
Sep 09 21:26:44.291348 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:44.291315    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzvkf_b3ab512a-1904-4284-82da-8a9a0a2aa4e1/node-ca/0.log"
Sep 09 21:26:45.602973 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:45.602943    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d7ff7ffd8-b545h_681323ae-d7bd-43ce-8bc6-34b06c1df3b8/router/0.log"
Sep 09 21:26:46.220065 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:46.220038    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qdgdx_5739f528-c67f-4c93-807e-80854f5b8b06/serve-healthcheck-canary/0.log"
Sep 09 21:26:46.829533 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:46.829486    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b7dkr_e6d164f9-1387-4545-9439-f10ecaa8b1a2/kube-rbac-proxy/0.log"
Sep 09 21:26:46.854434 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:46.854409    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b7dkr_e6d164f9-1387-4545-9439-f10ecaa8b1a2/exporter/0.log"
Sep 09 21:26:46.883683 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:46.883650    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b7dkr_e6d164f9-1387-4545-9439-f10ecaa8b1a2/extractor/0.log"
Sep 09 21:26:48.673035 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:48.673006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sghw5/perf-node-gather-daemonset-r4kt8"
Sep 09 21:26:50.141241 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:50.141197    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7f7677c75-wvxsr_8720a49b-11fe-48e3-b46b-34283524c366/manager/0.log"
Sep 09 21:26:50.177127 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:50.177073    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5fq7q_f386a589-3e94-4453-a100-b49fc67952f1/openshift-lws-operator/0.log"
Sep 09 21:26:50.789269 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:50.789238    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d7ff7ffd8-b545h_681323ae-d7bd-43ce-8bc6-34b06c1df3b8/router/0.log"
Sep 09 21:26:51.417979 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:51.417947    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-75f754565b-jzxdx_d3e47588-3ef6-4fcc-b2ed-813cd9001713/manager/0.log"
Sep 09 21:26:51.442724 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:51.442698    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-svwj5_310bdb08-7715-4a7f-baea-c03fdcc6795c/server/0.log"
Sep 09 21:26:52.115102 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:52.115053    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d7ff7ffd8-b545h_681323ae-d7bd-43ce-8bc6-34b06c1df3b8/router/0.log"
Sep 09 21:26:52.705744 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:52.705656    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-wrzkv_9e9e9ccc-16da-46ab-8bf3-baae89a73785/s3-init/0.log"
Sep 09 21:26:52.724108 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:52.724057    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-wrzkv_9e9e9ccc-16da-46ab-8bf3-baae89a73785/download-hf-model/0.log"
Sep 09 21:26:57.510783 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:57.510698    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-snf2g_9207419b-8115-4a5d-8c9c-5e10a5f6e52d/migrator/0.log"
Sep 09 21:26:57.536751 ip-10-0-131-185 kubenswrapper[2566]: I0909 21:26:57.536719    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-snf2g_9207419b-8115-4a5d-8c9c-5e10a5f6e52d/graceful-termination/0.log"