Sep 17 19:04:14.625745 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 19:04:14.625758 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 19:04:14.625768 ip-10-0-128-131 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 19:04:14.626077 ip-10-0-128-131 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 19:04:25.956525 ip-10-0-128-131 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 19:04:25.956540 ip-10-0-128-131 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 51a4e9ed5f03426ca6cd4c47869a7a0c --
Sep 17 19:06:36.449966 ip-10-0-128-131 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 19:06:40.019623 ip-10-0-128-131 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:40.019623 ip-10-0-128-131 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 19:06:40.019623 ip-10-0-128-131 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:40.019623 ip-10-0-128-131 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 19:06:40.019623 ip-10-0-128-131 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:40.020312 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.019712    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 19:06:40.024067 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024050    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:40.024067 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024067    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024071    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024074    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024077    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024081    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024084    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024087    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024090    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024093    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024096    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024098    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024101    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024104    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024106    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024114    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024117    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024120    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024122    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024125    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024128    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:40.024137 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024130    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024133    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024135    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024138    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024142    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024145    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024148    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024150    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024153    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024155    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024158    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024161    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024163    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024166    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024169    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024171    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024174    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024176    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024179    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024182    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:40.024611 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024184    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024187    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024189    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024192    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024195    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024198    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024200    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024203    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024206    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024208    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024212    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024216    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024219    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024222    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024225    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024229    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024233    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024236    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024239    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:40.025149 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024243    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024246    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024249    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024251    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024254    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024256    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024259    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024261    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024265    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024268    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024271    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024274    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024277    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024280    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024283    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024286    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024289    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024291    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024294    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024296    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:40.025608 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024299    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024301    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024304    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024306    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024309    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024311    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024314    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024319    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024322    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024324    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024327    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024329    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024332    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024335    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024337    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024339    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024342    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024345    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.024347    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:40.026120 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025077    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025085    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025088    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025091    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025094    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025097    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025099    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025102    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025105    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025107    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025110    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025113    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025115    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025118    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025120    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025122    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025125    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025127    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025130    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025133    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:40.026568 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025137    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025140    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025142    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025145    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025148    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025151    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025154    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025158    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025162    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025165    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025168    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025170    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025173    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025176    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025179    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025182    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025184    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025187    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025190    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:40.027078 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025193    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025196    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025199    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025201    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025204    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025206    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025209    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025211    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025214    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025216    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025219    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025221    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025224    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025228    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025230    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025233    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025236    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025238    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025241    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025243    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:40.027540 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025246    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025248    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025251    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025253    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025255    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025259    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025262    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025266    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025268    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025271    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025274    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025277    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025280    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025283    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025286    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025289    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025291    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025294    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025296    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:40.028087 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025299    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025301    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025304    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025306    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025308    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025311    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025314    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025317    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025319    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025322    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025325    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025327    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025330    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025332    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025335    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025338    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025342    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025344    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025347    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:40.028556 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025349    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.025352    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026494    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026502    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026506    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026509    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026513    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026516    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026519    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026522    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026524    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026527    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026530    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026532    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026535    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026538    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026541    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026544    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026547    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026550    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:40.029025 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026553    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026555    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026558    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026561    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026563    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026566    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026568    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026571    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026574    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026576    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026579    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026582    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026584    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026587    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026589    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026592    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026594    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026597    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026599    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026602    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:40.029521 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026604    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026607    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026610    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026612    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026615    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026617    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026620    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026623    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026625    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026628    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026631    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026634    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026637    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026640    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026642    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026645    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026647    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026651    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026655    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026657    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:40.030065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026660    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026662    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026665    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026667    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026671    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026675    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026677    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026680    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026696    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026699    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026703    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026705    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026708    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026711    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026714    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026716    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026719    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026721    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026724    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:40.030561 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026726    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026729    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026733    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026739    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026742    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026745    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026748    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026750    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026754    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026756    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026759    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026761    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026764    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026767    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026770    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026772    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026775    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026777    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026780    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:40.031066 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026783    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026785    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.026788    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026861    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026892    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026901    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026908    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026914    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026917    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026922    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026927    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026930    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026933    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026936    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026939    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026946    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026951    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026954    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026957    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026960    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026963    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026967    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026970    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026973    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 19:06:40.031527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026976    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026979    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026983    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026986    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026989    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026992    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026995    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.026998    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027001    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027004    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027006    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027010    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027013    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027016    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027019    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027021    2583 flags.go:64] FLAG: --enable-server="true"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027024    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027028    2583 flags.go:64] FLAG: --event-burst="100"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027032    2583 flags.go:64] FLAG: --event-qps="50"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027034    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027037    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027040    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027044    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027048    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027052    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 19:06:40.032133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027055    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027058    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027061    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027064    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027066    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027069    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027072    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027075    2583 flags.go:64] FLAG: --feature-gates=""
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027079    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027082    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027085    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027089    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027092    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027095    2583 flags.go:64] FLAG: --help="false"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027098    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027101    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027104    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027107    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027110    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027113    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027116    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027119    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027122    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 19:06:40.032747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027124    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027127    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027130    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027134    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027136    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027139    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027142    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027146    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027150    2583 flags.go:64] FLAG: --lock-file=""
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027153    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027156    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027159    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027164    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027167    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027169    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027172    2583 flags.go:64] FLAG: --logging-format="text"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027175    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027178    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027181    2583 flags.go:64] FLAG: --manifest-url=""
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027184    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027188    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027191    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027195    2583 flags.go:64] FLAG: --max-pods="110"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027197    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027201    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 19:06:40.033344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027203    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027207    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027209    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027212    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027215    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027222    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027224    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027227    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027230    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027233    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027238    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027241    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027244    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027247    2583 flags.go:64] FLAG: --port="10250"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027250    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027255    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0756a4a4b866aa73b"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027258    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027261    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027264    2583 flags.go:64] FLAG: --register-node="true"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027267    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027271    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027274    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027277    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027280    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027284    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 19:06:40.034034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027287    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027289    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027292    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027295    2583 flags.go:64] FLAG: --runonce="false"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027298    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027301    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027304    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027306    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027309    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027312    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027315    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027318    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027321    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027324    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027327    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027330    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027333    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027336    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027339    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027345    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027348    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027351    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027359    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027362    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027365    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 19:06:40.034776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027368    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027370    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027373    2583 flags.go:64] FLAG: --v="2"
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027377    2583 flags.go:64] FLAG: --version="false"
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027381    2583 flags.go:64] FLAG: --vmodule=""
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027384    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.027387    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027524    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027528    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027531    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027533    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027536    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027539    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027541    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027544    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027547    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027549    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027552    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027555    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027557    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027560    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027562    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:40.035478 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027565    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027576    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027579    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027582    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027584    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027588    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027590    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027596    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027599    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027601    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027604    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027606    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027609    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027611    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027614    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027617    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027619    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027622    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027624    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027627    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:40.036117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027629    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027632    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027634    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027637    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027639    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027642    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027645    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027649    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027652    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027655    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027658    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027661    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027664    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027666    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027669    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027671    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027674    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027676    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027679    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027698    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:40.036791 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027701    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027704    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027706    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027709    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027712    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027714    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027717    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027719    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027722    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027725    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027727    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027729    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027732    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027735    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027737    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027740    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027743    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027745    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027748    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027750    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:40.037653 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027754    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027758    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027762    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027765    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027768    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027771    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027773    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027776    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027778    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027781    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027783    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027789    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027791    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027794    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027797    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027799    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027802    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027804    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027807    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:40.038325 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027810    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027812    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027815    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027817    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.027820    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.028364    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.037936    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.037953    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.037973    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.037979    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038042    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038047    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038050    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038053    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:40.038813 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038056    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038059    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038062    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038064    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038067    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038069    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038072    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038075    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038085    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038088    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038091    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038094    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038097    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038100    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038102    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038105    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038107    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038110    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038113    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038115    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:40.039163 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038118    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038120    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038123    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038129    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038132    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038135    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038138    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038140    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038144    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038147    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038149    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038152    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038156    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038160    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038163    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038166    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038169    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038172    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038175    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038177    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:40.039635 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038180    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038183    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038185    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038188    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038191    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038193    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038196    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038198    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038201    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038203    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038206    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038210    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038214    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038217    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038219    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038223    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038226    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038229    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038232    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:40.040132 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038235    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038238    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038240    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038243    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038246    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038248    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038251    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038253    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038256    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038259    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038261    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038264    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038267    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038269    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038272    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038274    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038279    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038282    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038285    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038287    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:40.040595 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038290    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038295    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038298    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038300    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038303    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038305    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038308    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038310    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038313    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038316    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038318    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038321    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038324    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038327    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:40.041218 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038329    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.038334    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038449    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038454    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038457    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038460    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038464    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038466    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038469    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038472    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038474    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038477    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038480    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038483    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038486    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:40.041569 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038488    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038491    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038494    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038496    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038499    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038501    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038504    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038506    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038509    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038512    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038516    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038519    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038522    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038526    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038529    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038531    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038534    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038536    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038539    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:40.042033 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038541    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038544    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038546    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038549    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038551    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038555    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038559    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038561    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038564    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038567    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038570    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038573    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038576    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038579    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038583    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038586    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038589    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038592    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038595    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:40.042843 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038597    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038600    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038602    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038605    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038607    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038610    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038613    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038616    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038618    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038621    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038623    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038626    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038628    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038631    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038633    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038635    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038638    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038640    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038643    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038645    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038648    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:40.043430 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038650    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038653    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038655    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038658    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038660    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038663    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038666    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038669    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038671    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038674    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038676    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038679    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038681    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038704    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038707    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038710    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038712    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038715    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038718    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038721    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:40.044098 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038723    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038726    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038729    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038731    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038734    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038737    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:40.038739    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.038744    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.038846    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.040778    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.041804    2583 server.go:1019] "Starting client certificate rotation"
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.041919    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.041962    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:40.044622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.044271    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:40.045456 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.045438    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:40.051481 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.051465    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 17 19:06:40.053018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.053005    2583 log.go:25] "Validated CRI v1 image API"
Sep 17 19:06:40.053869 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.053855    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 19:06:40.056182 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.056158    2583 fs.go:135] Filesystem UUIDs: map[3dc50013-3daf-4e37-8fe4-99283affafa6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9e42b540-38a6-486a-a14a-e25c7717b4bb:/dev/nvme0n1p4]
Sep 17 19:06:40.056262 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.056181    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 19:06:40.060753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.060632    2583 manager.go:216] Machine: {Timestamp:2026-09-17 19:06:40.060167271 +0000 UTC m=+2.108047118 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101178 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27590e231bad8ded36e241be9ba553 SystemUUID:ec27590e-231b-ad8d-ed36-e241be9ba553 BootID:51a4e9ed-5f03-426c-a6cd-4c47869a7a0c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ca:c1:1d:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ca:c1:1d:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:f8:24:56:43:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 19:06:40.060753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.060749    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 19:06:40.060852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.060818    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 19:06:40.062507 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.062477    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 19:06:40.062661 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.062509    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 19:06:40.063922 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.063909    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 19:06:40.063976 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.063930    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 19:06:40.063976 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.063946    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:40.063976 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.063962    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:40.064106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.064047    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 19:06:40.067957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.067944    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:40.068085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068075    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 19:06:40.068139 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068116    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 19:06:40.068139 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068133    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 19:06:40.068237 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068150    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 19:06:40.068237 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068164    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 17 19:06:40.068237 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068175    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 19:06:40.068522 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068507    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:40.068579 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.068546    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:40.072132 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.072113    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 19:06:40.078503 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.078478    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:40.079572 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.079554    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 19:06:40.079651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.079584    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 19:06:40.080730 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080714    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 19:06:40.080730 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080733    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080742    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080749    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080759    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080764    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080778    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080783    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080790    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080795    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080803    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080808    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080836    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 19:06:40.080839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.080841    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 19:06:40.081899 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.081877    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 19:06:40.081968 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.081880    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 19:06:40.084738 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.084719    2583 server.go:1268] "Started kubelet"
Sep 17 19:06:40.084827 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.084745    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 19:06:40.084827 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.084787    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 19:06:40.084916 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.084879    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 19:06:40.084947 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.084932    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 19:06:40.085600 ip-10-0-128-131 systemd[1]: Started Kubernetes Kubelet.
Sep 17 19:06:40.086240 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.085759    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 19:06:40.086301 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.086285    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 19:06:40.089612 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.089595    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 19:06:40.092731 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.092713    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 19:06:40.092817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.092738    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:40.092817 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.092811    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.093294 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093266    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 19:06:40.093294 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093285    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 19:06:40.093436 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093269    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 19:06:40.093436 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093406    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 19:06:40.093436 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093413    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 19:06:40.093554 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093517    2583 factory.go:55] Registering systemd factory
Sep 17 19:06:40.093554 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093532    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 17 19:06:40.093819 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093799    2583 factory.go:153] Registering CRI-O factory
Sep 17 19:06:40.093878 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093822    2583 factory.go:223] Registration of the crio container factory successfully
Sep 17 19:06:40.093925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093881    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 19:06:40.093925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093902    2583 factory.go:103] Registering Raw factory
Sep 17 19:06:40.093925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.093918    2583 manager.go:1196] Started watching for new ooms in manager
Sep 17 19:06:40.094336 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.094322    2583 manager.go:349] Starting recovery of all containers
Sep 17 19:06:40.095915 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.095888    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 19:06:40.096031 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.096002    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 19:06:40.135008 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.134976    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6l8m8"
Sep 17 19:06:40.135008 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.135008    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6l8m8"
Sep 17 19:06:40.135614 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.135592    2583 manager.go:354] Recovery completed
Sep 17 19:06:40.141306 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.141291    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:40.143145 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.143122    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:40.143222 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.143152    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:40.143222 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.143162    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:40.143729 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.143714    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 19:06:40.143729 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.143728    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 19:06:40.143820 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.143748    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:40.143820 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.143729    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:40.144533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.144520    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:40.144580 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.144545    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:40.144580 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.144555    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:40.145753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.145738    2583 policy_none.go:49] "None policy: Start"
Sep 17 19:06:40.145753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.145756    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 19:06:40.145843 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.145768    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 19:06:40.147560 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.147548    2583 policy_none.go:47] "Start"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.192867    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.193789    2583 manager.go:337] "Starting Device Plugin manager"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.193819    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.193828    2583 server.go:87] "Starting device plugin registration server"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.194089    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.194112    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.194211    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.194294    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 19:06:40.194578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.194304    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 19:06:40.196020 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.195994    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 19:06:40.196119 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.196055    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.247046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.247003    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 19:06:40.248455 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.248428    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 19:06:40.248575 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.248497    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 19:06:40.248575 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.248525    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 19:06:40.248705 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.248578    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 19:06:40.250950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.250927    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:40.294301 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.294217    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:40.295196 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.295176    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:40.295296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.295216    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:40.295296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.295227    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:40.295296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.295250    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.305168 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.305148    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.305168 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.305168    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-131.ec2.internal\": node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.338763 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.338743    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.348805 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.348785    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"]
Sep 17 19:06:40.348876 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.348866    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:40.350385 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.350371    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:40.350446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.350399    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:40.350446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.350409    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:40.352449 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.352437    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:40.352559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.352545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.352608 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.352573    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:40.353139 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.353124    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:40.353221 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.353155    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:40.353221 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.353166    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:40.353221 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.353129    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:40.353357 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.353239    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:40.353357 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.353257    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:40.355208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.355198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.355274 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.355217    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:40.355919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.355904    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:40.355977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.355934    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:40.355977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.355945    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:40.383361 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.383343    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.387595 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.387579    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-131.ec2.internal\" not found" node="ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.394787 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.394770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0c8283cc05ebedf3cccae5e2687eb60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"e0c8283cc05ebedf3cccae5e2687eb60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.394857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.394796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c8283cc05ebedf3cccae5e2687eb60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"e0c8283cc05ebedf3cccae5e2687eb60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.394857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.394812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e72bea0306cf2b83af0585e51b861b54-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"e72bea0306cf2b83af0585e51b861b54\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.439062 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.439044    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.495841 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.495816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e72bea0306cf2b83af0585e51b861b54-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"e72bea0306cf2b83af0585e51b861b54\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.495841 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.495842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0c8283cc05ebedf3cccae5e2687eb60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"e0c8283cc05ebedf3cccae5e2687eb60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.496000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.495858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c8283cc05ebedf3cccae5e2687eb60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"e0c8283cc05ebedf3cccae5e2687eb60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.496000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.495884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c8283cc05ebedf3cccae5e2687eb60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"e0c8283cc05ebedf3cccae5e2687eb60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.496000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.495919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0c8283cc05ebedf3cccae5e2687eb60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal\" (UID: \"e0c8283cc05ebedf3cccae5e2687eb60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.496000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.495916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e72bea0306cf2b83af0585e51b861b54-config\") pod \"kube-apiserver-proxy-ip-10-0-128-131.ec2.internal\" (UID: \"e72bea0306cf2b83af0585e51b861b54\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.539526 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.539505    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.639927 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.639859    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.687070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.687037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.690911 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:40.690889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:40.740636 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.740595    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.841056 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.841017    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:40.941633 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:40.941569    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:41.042085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.042053    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 19:06:41.042085 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:41.042075    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:41.042528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.042209    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:41.141325 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.141282    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 19:01:40 +0000 UTC" deadline="2028-03-07 01:44:16.89935108 +0000 UTC"
Sep 17 19:06:41.141325 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.141320    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12870h37m35.758034731s"
Sep 17 19:06:41.142378 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:41.142359    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:41.242950 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:41.242885    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:41.290392 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:41.290362    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72bea0306cf2b83af0585e51b861b54.slice/crio-860f2cced4920ed407a5411a8650237625170094e5b7b1b94c3acc8a36cc780b WatchSource:0}: Error finding container 860f2cced4920ed407a5411a8650237625170094e5b7b1b94c3acc8a36cc780b: Status 404 returned error can't find the container with id 860f2cced4920ed407a5411a8650237625170094e5b7b1b94c3acc8a36cc780b
Sep 17 19:06:41.290793 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:41.290766    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c8283cc05ebedf3cccae5e2687eb60.slice/crio-518aa9f24e0cf68de2a69d89d21fbcdbb14ebeedbfb5b555f8b0f0dcc153a67d WatchSource:0}: Error finding container 518aa9f24e0cf68de2a69d89d21fbcdbb14ebeedbfb5b555f8b0f0dcc153a67d: Status 404 returned error can't find the container with id 518aa9f24e0cf68de2a69d89d21fbcdbb14ebeedbfb5b555f8b0f0dcc153a67d
Sep 17 19:06:41.297486 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.297469    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:06:41.343955 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:41.343933    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:41.396772 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.396750    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:41.444395 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:41.444361    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:41.474220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.474205    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:41.544765 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:41.544707    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-131.ec2.internal\" not found"
Sep 17 19:06:41.602870 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.602849    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:41.693516 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.693491    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:41.702052 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.702034    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:41.703049 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.703017    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal"
Sep 17 19:06:41.715574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:41.715555    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:42.068517 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.068471    2583 apiserver.go:52] "Watching apiserver"
Sep 17 19:06:42.077600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.077023    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 19:06:42.078424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.078393    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal","openshift-multus/multus-qjbj4","openshift-multus/network-metrics-daemon-hpls8","kube-system/global-pull-secret-syncer-tl6xh","kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6","openshift-image-registry/node-ca-qrd25","openshift-multus/multus-additional-cni-plugins-zz8lm","openshift-network-diagnostics/network-check-target-d5lhg","openshift-network-operator/iptables-alerter-q5d75","openshift-ovn-kubernetes/ovnkube-node-vgdhs","kube-system/konnectivity-agent-4qvt9","openshift-cluster-node-tuning-operator/tuned-qqgr9","openshift-dns/node-resolver-njvgl"]
Sep 17 19:06:42.081047 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.081025    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.083517 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.083367    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gpm2r\""
Sep 17 19:06:42.085036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.084234    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 19:06:42.085036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.084234    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 19:06:42.085036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.084301    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 19:06:42.085036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.084423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.086709 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.086221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:42.086709 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.086286    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:42.086852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.086725    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 19:06:42.086852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.086732    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 19:06:42.088357 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.088337    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.088444 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.088400    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:42.089391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.088908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 19:06:42.089391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.088937    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 19:06:42.089391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.088987    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 19:06:42.089391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.089164    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 19:06:42.089391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.089220    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nrsff\""
Sep 17 19:06:42.091006 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.090986    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.093194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.093175    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.094441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.093563    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8jv8k\""
Sep 17 19:06:42.094441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.093739    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 19:06:42.094441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.093826    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 19:06:42.094441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.093958    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:42.095840 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.095719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:42.095840 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.095779    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:42.095994 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.095974    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 19:06:42.096727 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.096709    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 19:06:42.097090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.096979    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8mxc8\""
Sep 17 19:06:42.097090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.096869    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 19:06:42.097795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.097776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.101472 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.100652    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dk7l2\""
Sep 17 19:06:42.101472 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.100778    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:42.101472 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.100803    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:42.101472 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.100996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.102737 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.102715    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.104021 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.104003    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 19:06:42.104250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.104234    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 19:06:42.104456 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.104441    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 19:06:42.105291 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.105264    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6zldl\""
Sep 17 19:06:42.106363 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.106348    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 19:06:42.106544 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.106527    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:42.106768 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.106752    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:42.107125 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.107109    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cf44v\""
Sep 17 19:06:42.107355 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.107338    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 19:06:42.107432 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.107420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.108006 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.107987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.108081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-env-overrides\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.108081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e17c207-4488-49ec-b117-644ad958d38c-hosts-file\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.108081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e17c207-4488-49ec-b117-644ad958d38c-tmp-dir\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.108226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovnkube-config\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.108226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68nw\" (UniqueName: \"kubernetes.io/projected/35716ede-fc12-454a-a045-aca7fee96b1e-kube-api-access-s68nw\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:42.108226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-kubelet\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.108226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-slash\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.108226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-run-netns\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.108226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-etc-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.108562 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d065b95-a94c-4a9f-a0b1-5f20f660c173-dbus\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.108562 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.108495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.109123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-host\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.109199 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-serviceca\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.109199 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovnkube-script-lib\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:42.109302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.109302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-socket-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.109302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-device-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.109489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfgc\" (UniqueName: \"kubernetes.io/projected/9280330f-ef46-4bf3-84ed-7273c5f7ad58-kube-api-access-qdfgc\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.109489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-systemd-units\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-cni-netd\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpzlc\" (UniqueName: \"kubernetes.io/projected/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-kube-api-access-qpzlc\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2r7\" (UniqueName: \"kubernetes.io/projected/9e17c207-4488-49ec-b117-644ad958d38c-kube-api-access-9l2r7\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.109744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovn-node-metrics-cert\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d065b95-a94c-4a9f-a0b1-5f20f660c173-kubelet-config\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.109744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-registration-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.109744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-systemd\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109722    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 19:06:42.109950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-ovn\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-log-socket\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.109950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsfff\" (UniqueName: \"kubernetes.io/projected/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-kube-api-access-qsfff\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.109950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-var-lib-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.110095 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.109942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.110134 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-cni-bin\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.110180 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gn5c9\""
Sep 17 19:06:42.110180 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.110256 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:42.110256 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-node-log\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.110335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110257    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.110335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-sys-fs\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.110412 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110353    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:42.110642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110624    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 19:06:42.110719 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 19:06:42.110887 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110869    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 19:06:42.110969 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.110896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ngrh4\""
Sep 17 19:06:42.140818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.140777    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-45s2x"
Sep 17 19:06:42.149703 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.148500    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-45s2x"
Sep 17 19:06:42.194681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.194653    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 19:06:42.211250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-var-lib-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:42.211384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211281    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-var-lib-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-kubernetes\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.211384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-system-cni-dir\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.211384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-cni-bin\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.211384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-node-log\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-run\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-host\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211436    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-node-log\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-system-cni-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-etc-kubernetes\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-env-overrides\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e17c207-4488-49ec-b117-644ad958d38c-hosts-file\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e17c207-4488-49ec-b117-644ad958d38c-tmp-dir\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-cni-multus\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e17c207-4488-49ec-b117-644ad958d38c-hosts-file\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-daemon-config\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-slash\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-run-netns\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-etc-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.211957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfgc\" (UniqueName: \"kubernetes.io/projected/9280330f-ef46-4bf3-84ed-7273c5f7ad58-kube-api-access-qdfgc\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.211973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-systemd\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e17c207-4488-49ec-b117-644ad958d38c-tmp-dir\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-tuned\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-env-overrides\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovnkube-script-lib\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-slash\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-etc-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212210    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-run-netns\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-socket-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-device-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.212286    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-sys\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-device-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-socket-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.212361 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.212357    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs podName:35716ede-fc12-454a-a045-aca7fee96b1e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.712336786 +0000 UTC m=+4.760216623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs") pod "network-metrics-daemon-hpls8" (UID: "35716ede-fc12-454a-a045-aca7fee96b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/090fea3b-5c35-4a75-84b6-ff87bab92ef3-tmp\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhvp6\" (UniqueName: \"kubernetes.io/projected/8e76a145-f19d-4439-99c3-3501aae73b8d-kube-api-access-lhvp6\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-systemd-units\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-cni-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094a9e6e-f49f-4b2b-bed0-b49e090d090d-cni-binary-copy\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-systemd-units\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovn-node-metrics-cert\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.212985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.212670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d065b95-a94c-4a9f-a0b1-5f20f660c173-kubelet-config\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d065b95-a94c-4a9f-a0b1-5f20f660c173-kubelet-config\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213025    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovnkube-script-lib\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-ovn\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-cnibin\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-ovn\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-multus-certs\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.213297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-cni-bin\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-cni-bin\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-openvswitch\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b0adfc-769a-43b7-975d-e84c8603d651-iptables-alerter-script\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-socket-dir-parent\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-sys-fs\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khktb\" (UniqueName: \"kubernetes.io/projected/55b0adfc-769a-43b7-975d-e84c8603d651-kube-api-access-khktb\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysctl-d\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-sys-fs\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.213666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-cnibin\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b0adfc-769a-43b7-975d-e84c8603d651-host-slash\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovnkube-config\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68nw\" (UniqueName: \"kubernetes.io/projected/35716ede-fc12-454a-a045-aca7fee96b1e-kube-api-access-s68nw\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-kubelet\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d065b95-a94c-4a9f-a0b1-5f20f660c173-dbus\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysctl-conf\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-lib-modules\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.213964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-host\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-serviceca\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f016d2b8-1a3f-47ee-885a-8b43bdd1383e-agent-certs\") pod \"konnectivity-agent-4qvt9\" (UID: \"f016d2b8-1a3f-47ee-885a-8b43bdd1383e\") " pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysconfig\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovnkube-config\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.214299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-netns\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-cni-netd\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpzlc\" (UniqueName: \"kubernetes.io/projected/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-kube-api-access-qpzlc\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2r7\" (UniqueName: \"kubernetes.io/projected/9e17c207-4488-49ec-b117-644ad958d38c-kube-api-access-9l2r7\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-modprobe-d\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-var-lib-kubelet\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6v66\" (UniqueName: \"kubernetes.io/projected/090fea3b-5c35-4a75-84b6-ff87bab92ef3-kube-api-access-k6v66\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-registration-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214453    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-k8s-cni-cncf-io\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-kubelet\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-hostroot\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-conf-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrtq\" (UniqueName: \"kubernetes.io/projected/094a9e6e-f49f-4b2b-bed0-b49e090d090d-kube-api-access-xnrtq\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-systemd\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-log-socket\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f016d2b8-1a3f-47ee-885a-8b43bdd1383e-konnectivity-ca\") pod \"konnectivity-agent-4qvt9\" (UID: \"f016d2b8-1a3f-47ee-885a-8b43bdd1383e\") " pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.215086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsfff\" (UniqueName: \"kubernetes.io/projected/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-kube-api-access-qsfff\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-host\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-kubelet\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.214978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.215017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-host-cni-netd\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.215189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-registration-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.215274    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d065b95-a94c-4a9f-a0b1-5f20f660c173-dbus\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.215330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-run-systemd\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.215354    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-log-socket\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.215365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9280330f-ef46-4bf3-84ed-7273c5f7ad58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.215482    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.215562    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret podName:0d065b95-a94c-4a9f-a0b1-5f20f660c173 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.715544796 +0000 UTC m=+4.763424633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret") pod "global-pull-secret-syncer-tl6xh" (UID: "0d065b95-a94c-4a9f-a0b1-5f20f660c173") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.215961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.215826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-serviceca\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.216766 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.216710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-ovn-node-metrics-cert\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.222555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.222511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfgc\" (UniqueName: \"kubernetes.io/projected/9280330f-ef46-4bf3-84ed-7273c5f7ad58-kube-api-access-qdfgc\") pod \"aws-ebs-csi-driver-node-ltxm6\" (UID: \"9280330f-ef46-4bf3-84ed-7273c5f7ad58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.223501 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.223319    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:42.223501 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.223343    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:42.223501 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.223357    2583 projected.go:196] Error preparing data for projected volume kube-api-access-5fsqw for pod openshift-network-diagnostics/network-check-target-d5lhg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:42.223501 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.223418    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw podName:7b583cdf-d259-4eae-b3fb-f929b34cd821 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.723402634 +0000 UTC m=+4.771282469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5fsqw" (UniqueName: "kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw") pod "network-check-target-d5lhg" (UID: "7b583cdf-d259-4eae-b3fb-f929b34cd821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:42.226153 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.226088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsfff\" (UniqueName: \"kubernetes.io/projected/6efce3ed-bfc7-4d90-bc0c-af87cba720c8-kube-api-access-qsfff\") pod \"node-ca-qrd25\" (UID: \"6efce3ed-bfc7-4d90-bc0c-af87cba720c8\") " pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.227451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.226912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpzlc\" (UniqueName: \"kubernetes.io/projected/d7eebd66-0d16-4c8b-855b-c53da26cc8ec-kube-api-access-qpzlc\") pod \"ovnkube-node-vgdhs\" (UID: \"d7eebd66-0d16-4c8b-855b-c53da26cc8ec\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.228057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.228034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68nw\" (UniqueName: \"kubernetes.io/projected/35716ede-fc12-454a-a045-aca7fee96b1e-kube-api-access-s68nw\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:42.228934 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.228910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2r7\" (UniqueName: \"kubernetes.io/projected/9e17c207-4488-49ec-b117-644ad958d38c-kube-api-access-9l2r7\") pod \"node-resolver-njvgl\" (UID: \"9e17c207-4488-49ec-b117-644ad958d38c\") " pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.253652 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.253606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" event={"ID":"e72bea0306cf2b83af0585e51b861b54","Type":"ContainerStarted","Data":"860f2cced4920ed407a5411a8650237625170094e5b7b1b94c3acc8a36cc780b"}
Sep 17 19:06:42.255174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.255147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"e0c8283cc05ebedf3cccae5e2687eb60","Type":"ContainerStarted","Data":"518aa9f24e0cf68de2a69d89d21fbcdbb14ebeedbfb5b555f8b0f0dcc153a67d"}
Sep 17 19:06:42.315579 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-kubernetes\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.315757 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-system-cni-dir\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.315757 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-cni-bin\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.315757 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-run\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.315757 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-system-cni-dir\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.315757 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-kubernetes\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.315757 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-cni-bin\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-host\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-system-cni-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-etc-kubernetes\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315830    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-run\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-cni-multus\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-daemon-config\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-host\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-systemd\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-cni-multus\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.315995    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-systemd\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-system-cni-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-etc-kubernetes\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-tuned\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-sys\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/090fea3b-5c35-4a75-84b6-ff87bab92ef3-tmp\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvp6\" (UniqueName: \"kubernetes.io/projected/8e76a145-f19d-4439-99c3-3501aae73b8d-kube-api-access-lhvp6\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-cni-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094a9e6e-f49f-4b2b-bed0-b49e090d090d-cni-binary-copy\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-cnibin\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-multus-certs\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-daemon-config\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b0adfc-769a-43b7-975d-e84c8603d651-iptables-alerter-script\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-socket-dir-parent\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khktb\" (UniqueName: \"kubernetes.io/projected/55b0adfc-769a-43b7-975d-e84c8603d651-kube-api-access-khktb\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysctl-d\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.316565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-cnibin\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.317389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b0adfc-769a-43b7-975d-e84c8603d651-host-slash\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.317389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.316572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.317389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.317389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-cni-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.317589 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-sys\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.317643 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/094a9e6e-f49f-4b2b-bed0-b49e090d090d-cni-binary-copy\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysctl-conf\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-lib-modules\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55b0adfc-769a-43b7-975d-e84c8603d651-host-slash\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysctl-d\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-multus-certs\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-cnibin\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317891    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-socket-dir-parent\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f016d2b8-1a3f-47ee-885a-8b43bdd1383e-agent-certs\") pod \"konnectivity-agent-4qvt9\" (UID: \"f016d2b8-1a3f-47ee-885a-8b43bdd1383e\") " pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e76a145-f19d-4439-99c3-3501aae73b8d-cnibin\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysconfig\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-lib-modules\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.317985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysconfig\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318000    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-netns\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.318627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-sysctl-conf\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-modprobe-d\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-var-lib-kubelet\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-netns\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6v66\" (UniqueName: \"kubernetes.io/projected/090fea3b-5c35-4a75-84b6-ff87bab92ef3-kube-api-access-k6v66\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-k8s-cni-cncf-io\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-modprobe-d\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/090fea3b-5c35-4a75-84b6-ff87bab92ef3-var-lib-kubelet\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-kubelet\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-hostroot\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-run-k8s-cni-cncf-io\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-conf-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318240    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-host-var-lib-kubelet\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrtq\" (UniqueName: \"kubernetes.io/projected/094a9e6e-f49f-4b2b-bed0-b49e090d090d-kube-api-access-xnrtq\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f016d2b8-1a3f-47ee-885a-8b43bdd1383e-konnectivity-ca\") pod \"konnectivity-agent-4qvt9\" (UID: \"f016d2b8-1a3f-47ee-885a-8b43bdd1383e\") " pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-hostroot\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/094a9e6e-f49f-4b2b-bed0-b49e090d090d-multus-conf-dir\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55b0adfc-769a-43b7-975d-e84c8603d651-iptables-alerter-script\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.319386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.318992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f016d2b8-1a3f-47ee-885a-8b43bdd1383e-konnectivity-ca\") pod \"konnectivity-agent-4qvt9\" (UID: \"f016d2b8-1a3f-47ee-885a-8b43bdd1383e\") " pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.320066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.319002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e76a145-f19d-4439-99c3-3501aae73b8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.320066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.319154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/090fea3b-5c35-4a75-84b6-ff87bab92ef3-etc-tuned\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.320066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.319283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/090fea3b-5c35-4a75-84b6-ff87bab92ef3-tmp\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.321672 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.321638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f016d2b8-1a3f-47ee-885a-8b43bdd1383e-agent-certs\") pod \"konnectivity-agent-4qvt9\" (UID: \"f016d2b8-1a3f-47ee-885a-8b43bdd1383e\") " pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.327930 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.327906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khktb\" (UniqueName: \"kubernetes.io/projected/55b0adfc-769a-43b7-975d-e84c8603d651-kube-api-access-khktb\") pod \"iptables-alerter-q5d75\" (UID: \"55b0adfc-769a-43b7-975d-e84c8603d651\") " pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.328551 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.328527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhvp6\" (UniqueName: \"kubernetes.io/projected/8e76a145-f19d-4439-99c3-3501aae73b8d-kube-api-access-lhvp6\") pod \"multus-additional-cni-plugins-zz8lm\" (UID: \"8e76a145-f19d-4439-99c3-3501aae73b8d\") " pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.328976 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.328956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrtq\" (UniqueName: \"kubernetes.io/projected/094a9e6e-f49f-4b2b-bed0-b49e090d090d-kube-api-access-xnrtq\") pod \"multus-qjbj4\" (UID: \"094a9e6e-f49f-4b2b-bed0-b49e090d090d\") " pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.329099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.329037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6v66\" (UniqueName: \"kubernetes.io/projected/090fea3b-5c35-4a75-84b6-ff87bab92ef3-kube-api-access-k6v66\") pod \"tuned-qqgr9\" (UID: \"090fea3b-5c35-4a75-84b6-ff87bab92ef3\") " pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.395907 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.395871    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrd25"
Sep 17 19:06:42.409829 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.409809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:06:42.424763 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.424735    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-njvgl"
Sep 17 19:06:42.430440 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.430412    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6"
Sep 17 19:06:42.440005 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.439987    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqgr9"
Sep 17 19:06:42.449555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.449533    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qjbj4"
Sep 17 19:06:42.458789 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.458756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q5d75"
Sep 17 19:06:42.468296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.468273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:06:42.476074 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.476044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zz8lm"
Sep 17 19:06:42.599048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.598968    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:42.721118 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.721083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:42.721312 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.721143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:42.721312 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.721265    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.721422 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.721339    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret podName:0d065b95-a94c-4a9f-a0b1-5f20f660c173 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:43.721316691 +0000 UTC m=+5.769196571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret") pod "global-pull-secret-syncer-tl6xh" (UID: "0d065b95-a94c-4a9f-a0b1-5f20f660c173") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.721704 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.721265    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.721751 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.721741    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs podName:35716ede-fc12-454a-a045-aca7fee96b1e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:43.721728592 +0000 UTC m=+5.769608427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs") pod "network-metrics-daemon-hpls8" (UID: "35716ede-fc12-454a-a045-aca7fee96b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.822419 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:42.822382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:42.822568 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.822549    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:42.822603 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.822576    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:42.822603 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.822586    2583 projected.go:196] Error preparing data for projected volume kube-api-access-5fsqw for pod openshift-network-diagnostics/network-check-target-d5lhg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:42.822671 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:42.822642    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw podName:7b583cdf-d259-4eae-b3fb-f929b34cd821 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:43.822627919 +0000 UTC m=+5.870507778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fsqw" (UniqueName: "kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw") pod "network-check-target-d5lhg" (UID: "7b583cdf-d259-4eae-b3fb-f929b34cd821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:43.152293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.152255    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:42 +0000 UTC" deadline="2028-02-17 11:42:26.605204285 +0000 UTC"
Sep 17 19:06:43.152293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.152288    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12424h35m43.452919686s"
Sep 17 19:06:43.176316 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.176067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094a9e6e_f49f_4b2b_bed0_b49e090d090d.slice/crio-7298a3d1e0ddb9a6d54db7ef7da66f58082c8a345163ce25fa51387b40078f26 WatchSource:0}: Error finding container 7298a3d1e0ddb9a6d54db7ef7da66f58082c8a345163ce25fa51387b40078f26: Status 404 returned error can't find the container with id 7298a3d1e0ddb9a6d54db7ef7da66f58082c8a345163ce25fa51387b40078f26
Sep 17 19:06:43.178462 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.178438    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b0adfc_769a_43b7_975d_e84c8603d651.slice/crio-a648ce643907336a9e2896ee6ee3e51369c85723469c1dd5aa1a86c5e5ef1d30 WatchSource:0}: Error finding container a648ce643907336a9e2896ee6ee3e51369c85723469c1dd5aa1a86c5e5ef1d30: Status 404 returned error can't find the container with id a648ce643907336a9e2896ee6ee3e51369c85723469c1dd5aa1a86c5e5ef1d30
Sep 17 19:06:43.200919 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.200890    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090fea3b_5c35_4a75_84b6_ff87bab92ef3.slice/crio-6977afb90c4c068dff29fd8c0b425afd3456d69cd529f78df6a61ee8d942082d WatchSource:0}: Error finding container 6977afb90c4c068dff29fd8c0b425afd3456d69cd529f78df6a61ee8d942082d: Status 404 returned error can't find the container with id 6977afb90c4c068dff29fd8c0b425afd3456d69cd529f78df6a61ee8d942082d
Sep 17 19:06:43.201746 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.201718    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9280330f_ef46_4bf3_84ed_7273c5f7ad58.slice/crio-0fce9348e3a66a5cb7199b7ebc04d33b43bcdc443f4494ca20aeaebaf8280383 WatchSource:0}: Error finding container 0fce9348e3a66a5cb7199b7ebc04d33b43bcdc443f4494ca20aeaebaf8280383: Status 404 returned error can't find the container with id 0fce9348e3a66a5cb7199b7ebc04d33b43bcdc443f4494ca20aeaebaf8280383
Sep 17 19:06:43.202575 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.202474    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e76a145_f19d_4439_99c3_3501aae73b8d.slice/crio-bf64c790692bc75f06b126e765795bb8bf82c14fc00e848f92742a005368212e WatchSource:0}: Error finding container bf64c790692bc75f06b126e765795bb8bf82c14fc00e848f92742a005368212e: Status 404 returned error can't find the container with id bf64c790692bc75f06b126e765795bb8bf82c14fc00e848f92742a005368212e
Sep 17 19:06:43.203019 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.202998    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efce3ed_bfc7_4d90_bc0c_af87cba720c8.slice/crio-cd4836174746f02f5f7289f9fdfc591cca9ff44b4b33504da9fb77d8f84ee8f8 WatchSource:0}: Error finding container cd4836174746f02f5f7289f9fdfc591cca9ff44b4b33504da9fb77d8f84ee8f8: Status 404 returned error can't find the container with id cd4836174746f02f5f7289f9fdfc591cca9ff44b4b33504da9fb77d8f84ee8f8
Sep 17 19:06:43.203818 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.203797    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7eebd66_0d16_4c8b_855b_c53da26cc8ec.slice/crio-fa60883e0255a1bdb903ec1848101e44743a32796f4fa9558c264d8cf2dedec2 WatchSource:0}: Error finding container fa60883e0255a1bdb903ec1848101e44743a32796f4fa9558c264d8cf2dedec2: Status 404 returned error can't find the container with id fa60883e0255a1bdb903ec1848101e44743a32796f4fa9558c264d8cf2dedec2
Sep 17 19:06:43.204643 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:06:43.204628    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf016d2b8_1a3f_47ee_885a_8b43bdd1383e.slice/crio-d6d0901967de902177f48135e91e264ed88e3ba4a731c798b0b480dbcb9a01d9 WatchSource:0}: Error finding container d6d0901967de902177f48135e91e264ed88e3ba4a731c798b0b480dbcb9a01d9: Status 404 returned error can't find the container with id d6d0901967de902177f48135e91e264ed88e3ba4a731c798b0b480dbcb9a01d9
Sep 17 19:06:43.258272 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.258232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqgr9" event={"ID":"090fea3b-5c35-4a75-84b6-ff87bab92ef3","Type":"ContainerStarted","Data":"6977afb90c4c068dff29fd8c0b425afd3456d69cd529f78df6a61ee8d942082d"}
Sep 17 19:06:43.259518 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.259488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qjbj4" event={"ID":"094a9e6e-f49f-4b2b-bed0-b49e090d090d","Type":"ContainerStarted","Data":"7298a3d1e0ddb9a6d54db7ef7da66f58082c8a345163ce25fa51387b40078f26"}
Sep 17 19:06:43.261011 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.260982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4qvt9" event={"ID":"f016d2b8-1a3f-47ee-885a-8b43bdd1383e","Type":"ContainerStarted","Data":"d6d0901967de902177f48135e91e264ed88e3ba4a731c798b0b480dbcb9a01d9"}
Sep 17 19:06:43.262387 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.262360    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"fa60883e0255a1bdb903ec1848101e44743a32796f4fa9558c264d8cf2dedec2"}
Sep 17 19:06:43.263743 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.263716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrd25" event={"ID":"6efce3ed-bfc7-4d90-bc0c-af87cba720c8","Type":"ContainerStarted","Data":"cd4836174746f02f5f7289f9fdfc591cca9ff44b4b33504da9fb77d8f84ee8f8"}
Sep 17 19:06:43.265109 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.265089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6" event={"ID":"9280330f-ef46-4bf3-84ed-7273c5f7ad58","Type":"ContainerStarted","Data":"0fce9348e3a66a5cb7199b7ebc04d33b43bcdc443f4494ca20aeaebaf8280383"}
Sep 17 19:06:43.266278 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.266255    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q5d75" event={"ID":"55b0adfc-769a-43b7-975d-e84c8603d651","Type":"ContainerStarted","Data":"a648ce643907336a9e2896ee6ee3e51369c85723469c1dd5aa1a86c5e5ef1d30"}
Sep 17 19:06:43.267402 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.267379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njvgl" event={"ID":"9e17c207-4488-49ec-b117-644ad958d38c","Type":"ContainerStarted","Data":"5b6efdeb43984608e04768cdfbc11110ee7e3ff98fdc00b3a4b9f7ef2b7d76d9"}
Sep 17 19:06:43.268421 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.268403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerStarted","Data":"bf64c790692bc75f06b126e765795bb8bf82c14fc00e848f92742a005368212e"}
Sep 17 19:06:43.728226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.727842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:43.728226 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.728070    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:43.728226 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.728136    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret podName:0d065b95-a94c-4a9f-a0b1-5f20f660c173 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.728118853 +0000 UTC m=+7.775998711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret") pod "global-pull-secret-syncer-tl6xh" (UID: "0d065b95-a94c-4a9f-a0b1-5f20f660c173") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:43.728226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.728174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:43.728526 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.728257    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:43.728526 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.728294    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs podName:35716ede-fc12-454a-a045-aca7fee96b1e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.728281642 +0000 UTC m=+7.776161478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs") pod "network-metrics-daemon-hpls8" (UID: "35716ede-fc12-454a-a045-aca7fee96b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:43.829991 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:43.829331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:43.829991 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.829537    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:43.829991 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.829561    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:43.829991 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.829573    2583 projected.go:196] Error preparing data for projected volume kube-api-access-5fsqw for pod openshift-network-diagnostics/network-check-target-d5lhg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:43.829991 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:43.829632    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw podName:7b583cdf-d259-4eae-b3fb-f929b34cd821 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:45.829614162 +0000 UTC m=+7.877494023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fsqw" (UniqueName: "kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw") pod "network-check-target-d5lhg" (UID: "7b583cdf-d259-4eae-b3fb-f929b34cd821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:44.152878 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.152789    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:42 +0000 UTC" deadline="2028-02-25 18:18:31.388845631 +0000 UTC"
Sep 17 19:06:44.152878 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.152831    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12623h11m47.236018357s"
Sep 17 19:06:44.252424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.251415    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:44.252424 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:44.251616    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:44.252424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.252120    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:44.252424 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:44.252234    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:44.252424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.252302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:44.252424 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:44.252375    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:44.274791 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.273895    2583 generic.go:356] "Generic (PLEG): container finished" podID="e0c8283cc05ebedf3cccae5e2687eb60" containerID="267b0933daa07ca0683b9e852f12574e8cd33093f950253821495856d86729b5" exitCode=0
Sep 17 19:06:44.274791 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.273995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"e0c8283cc05ebedf3cccae5e2687eb60","Type":"ContainerDied","Data":"267b0933daa07ca0683b9e852f12574e8cd33093f950253821495856d86729b5"}
Sep 17 19:06:44.288919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:44.288889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" event={"ID":"e72bea0306cf2b83af0585e51b861b54","Type":"ContainerStarted","Data":"b0a7a4a180439068911f3aee434974c5c1745a7b0fde9f9063442b15556faf39"}
Sep 17 19:06:45.313416 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:45.313376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" event={"ID":"e0c8283cc05ebedf3cccae5e2687eb60","Type":"ContainerStarted","Data":"ec504201a17aa4c4ac3fe25850d0eeae05e3c70418297cadc61a82f81ed2f46c"}
Sep 17 19:06:45.333064 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:45.332253    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-131.ec2.internal" podStartSLOduration=4.332228039 podStartE2EDuration="4.332228039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:44.31933042 +0000 UTC m=+6.367210277" watchObservedRunningTime="2026-09-17 19:06:45.332228039 +0000 UTC m=+7.380107899"
Sep 17 19:06:45.744670 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:45.744575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:45.744670 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:45.744646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:45.744904 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.744801    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:45.744904 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.744866    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret podName:0d065b95-a94c-4a9f-a0b1-5f20f660c173 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:49.74484736 +0000 UTC m=+11.792727200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret") pod "global-pull-secret-syncer-tl6xh" (UID: "0d065b95-a94c-4a9f-a0b1-5f20f660c173") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:45.745021 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.744937    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:45.745021 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.744975    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs podName:35716ede-fc12-454a-a045-aca7fee96b1e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:49.744962936 +0000 UTC m=+11.792842776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs") pod "network-metrics-daemon-hpls8" (UID: "35716ede-fc12-454a-a045-aca7fee96b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:45.845863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:45.845826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:45.846039 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.846017    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:45.846039 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.846036    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:45.846167 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.846056    2583 projected.go:196] Error preparing data for projected volume kube-api-access-5fsqw for pod openshift-network-diagnostics/network-check-target-d5lhg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:45.846167 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:45.846110    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw podName:7b583cdf-d259-4eae-b3fb-f929b34cd821 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:49.846093055 +0000 UTC m=+11.893972896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fsqw" (UniqueName: "kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw") pod "network-check-target-d5lhg" (UID: "7b583cdf-d259-4eae-b3fb-f929b34cd821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:46.249836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:46.249804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:46.250002 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:46.249940    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:46.250280 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:46.249804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:46.250395 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:46.250373    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:46.250451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:46.250439    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:46.250525 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:46.250507    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:48.249387 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:48.249352    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:48.249858 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:48.249491    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:48.249858 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:48.249832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:48.249962 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:48.249905    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:48.250067 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:48.250047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:48.250155 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:48.250137    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:49.786476 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:49.786421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:49.786969 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:49.786509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:49.786969 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.786630    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:49.786969 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.786714    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs podName:35716ede-fc12-454a-a045-aca7fee96b1e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.786674987 +0000 UTC m=+19.834554836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs") pod "network-metrics-daemon-hpls8" (UID: "35716ede-fc12-454a-a045-aca7fee96b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:49.787136 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.787092    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:49.787186 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.787156    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret podName:0d065b95-a94c-4a9f-a0b1-5f20f660c173 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.787143455 +0000 UTC m=+19.835023302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret") pod "global-pull-secret-syncer-tl6xh" (UID: "0d065b95-a94c-4a9f-a0b1-5f20f660c173") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:49.887940 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:49.887891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:49.888251 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.888073    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:49.888251 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.888109    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:49.888251 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.888123    2583 projected.go:196] Error preparing data for projected volume kube-api-access-5fsqw for pod openshift-network-diagnostics/network-check-target-d5lhg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:49.888251 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:49.888181    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw podName:7b583cdf-d259-4eae-b3fb-f929b34cd821 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:57.888165613 +0000 UTC m=+19.936045468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fsqw" (UniqueName: "kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw") pod "network-check-target-d5lhg" (UID: "7b583cdf-d259-4eae-b3fb-f929b34cd821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:50.251149 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:50.250176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:50.251149 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:50.250309    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:50.251149 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:50.250724    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:50.251149 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:50.250817    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:50.251149 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:50.250991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:50.251149 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:50.251087    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:52.249506 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:52.249473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:52.249992 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:52.249603    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:52.249992 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:52.249486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:52.249992 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:52.249731    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:52.249992 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:52.249485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:52.249992 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:52.249811    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:54.249297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:54.249264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:54.249761 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:54.249264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:54.249761 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:54.249402    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:54.249761 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:54.249472    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:54.249761 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:54.249264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:54.249761 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:54.249556    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:56.249231 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:56.249193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:56.249231 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:56.249222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:56.249722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:56.249193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:56.249722 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:56.249313    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:06:56.249722 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:56.249405    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:56.249722 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:56.249496    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:57.847659 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:57.847612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:57.848119 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:57.847710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:57.848119 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.847787    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:57.848119 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.847813    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:57.848119 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.847874    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs podName:35716ede-fc12-454a-a045-aca7fee96b1e nodeName:}" failed. No retries permitted until 2026-09-17 19:07:13.847855215 +0000 UTC m=+35.895735052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs") pod "network-metrics-daemon-hpls8" (UID: "35716ede-fc12-454a-a045-aca7fee96b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:57.848119 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.847894    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret podName:0d065b95-a94c-4a9f-a0b1-5f20f660c173 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:13.847884793 +0000 UTC m=+35.895764628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret") pod "global-pull-secret-syncer-tl6xh" (UID: "0d065b95-a94c-4a9f-a0b1-5f20f660c173") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:57.948651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:57.948608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:57.948847 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.948784    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:57.948847 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.948808    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:57.948847 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.948820    2583 projected.go:196] Error preparing data for projected volume kube-api-access-5fsqw for pod openshift-network-diagnostics/network-check-target-d5lhg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:57.949004 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:57.948887    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw podName:7b583cdf-d259-4eae-b3fb-f929b34cd821 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:13.948866732 +0000 UTC m=+35.996746570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fsqw" (UniqueName: "kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw") pod "network-check-target-d5lhg" (UID: "7b583cdf-d259-4eae-b3fb-f929b34cd821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:58.249205 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:58.249121    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:06:58.249205 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:58.249176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:06:58.249380 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:06:58.249183    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:06:58.249380 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:58.249268    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:06:58.249660 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:58.249629    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:06:58.249814 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:06:58.249746    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:00.249996 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:00.249956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:00.250419 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:00.250072    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:00.250419 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:00.250384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:00.250540 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:00.250468    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:00.250540 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:00.250510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:00.250645 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:00.250570    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:01.346993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.346715    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e76a145-f19d-4439-99c3-3501aae73b8d" containerID="5e827ecbf329e24bc67709fa3a871919ece5e6ac40b2bcb35a810c0e4bd3b22a" exitCode=0
Sep 17 19:07:01.347754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.346758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerDied","Data":"5e827ecbf329e24bc67709fa3a871919ece5e6ac40b2bcb35a810c0e4bd3b22a"}
Sep 17 19:07:01.348518 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.348495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqgr9" event={"ID":"090fea3b-5c35-4a75-84b6-ff87bab92ef3","Type":"ContainerStarted","Data":"83d413527f469b56778498686856a898eff629b33024fed2c7fc71c3fe068d6e"}
Sep 17 19:07:01.350070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.350050    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qjbj4" event={"ID":"094a9e6e-f49f-4b2b-bed0-b49e090d090d","Type":"ContainerStarted","Data":"d578a79e21a68abbd624bbb6fad76eadd37acfe5b771a8395385b1165a14768c"}
Sep 17 19:07:01.351452 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.351429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4qvt9" event={"ID":"f016d2b8-1a3f-47ee-885a-8b43bdd1383e","Type":"ContainerStarted","Data":"9d60fc25f240ee9132b0cf1f9def6fec3e1a679a395ed8afc660b4b5cb670a96"}
Sep 17 19:07:01.354082 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.354065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"dd6bfabf9feaf36eb9afb33be2dec5d25b881ae457636dda0714717862439682"}
Sep 17 19:07:01.354150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.354127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"15de40e84d997918eb04b513fbcd33d68487c061f0eb162b0101bcd10282f695"}
Sep 17 19:07:01.354194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.354170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"49b7c2687289e5b17896a062cdb1d5f66f5fe193b598bccbf51e43992964c275"}
Sep 17 19:07:01.354235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.354207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"eb550cedb14215769297d02fab8b54009e0df6d86281fd8ade99d77a4f0ae48e"}
Sep 17 19:07:01.355246 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.355230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrd25" event={"ID":"6efce3ed-bfc7-4d90-bc0c-af87cba720c8","Type":"ContainerStarted","Data":"c5fb2a9732437a924f61a836ddbb1cdedac68434fa2e894abb20777ed7783536"}
Sep 17 19:07:01.356461 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.356442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6" event={"ID":"9280330f-ef46-4bf3-84ed-7273c5f7ad58","Type":"ContainerStarted","Data":"a2b2266ac5dc45f7c0a8f38ab2be83cfe1715414dd537fdca380c6f6e0f1d0d5"}
Sep 17 19:07:01.357729 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.357710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-njvgl" event={"ID":"9e17c207-4488-49ec-b117-644ad958d38c","Type":"ContainerStarted","Data":"4e7c3122a6ddbcb082968f2c28474e2eb89b64f6c2b7c44c8b7067be8feba0ff"}
Sep 17 19:07:01.397488 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.397445    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-131.ec2.internal" podStartSLOduration=20.397432678 podStartE2EDuration="20.397432678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:45.332711697 +0000 UTC m=+7.380591555" watchObservedRunningTime="2026-09-17 19:07:01.397432678 +0000 UTC m=+23.445312535"
Sep 17 19:07:01.435367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.435324    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qjbj4" podStartSLOduration=3.708251066 podStartE2EDuration="21.435310883s" totalImagesPullingTime="17.727059817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.434885619 +0000 UTC m=+23.482765479" watchObservedRunningTime="2026-09-17 19:07:01.435310883 +0000 UTC m=+23.483190740"
Sep 17 19:07:01.451541 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.451498    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4qvt9" podStartSLOduration=3.82202823 podStartE2EDuration="21.451489531s" totalImagesPullingTime="17.629461301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.450986282 +0000 UTC m=+23.498866139" watchObservedRunningTime="2026-09-17 19:07:01.451489531 +0000 UTC m=+23.499369387"
Sep 17 19:07:01.470448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.470366    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-njvgl" podStartSLOduration=3.841986627 podStartE2EDuration="21.470355296s" totalImagesPullingTime="17.628368669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.470195676 +0000 UTC m=+23.518075534" watchObservedRunningTime="2026-09-17 19:07:01.470355296 +0000 UTC m=+23.518235152"
Sep 17 19:07:01.487916 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.487881    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qrd25" podStartSLOduration=11.235799788 podStartE2EDuration="21.48786985s" totalImagesPullingTime="10.252070062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.487532208 +0000 UTC m=+23.535412066" watchObservedRunningTime="2026-09-17 19:07:01.48786985 +0000 UTC m=+23.535749706"
Sep 17 19:07:01.507970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.507937    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qqgr9" podStartSLOduration=3.854364012 podStartE2EDuration="21.507926157s" totalImagesPullingTime="17.653562145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.506767714 +0000 UTC m=+23.554647570" watchObservedRunningTime="2026-09-17 19:07:01.507926157 +0000 UTC m=+23.555806014"
Sep 17 19:07:01.967813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:01.967792    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 19:07:02.145537 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.145470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:07:02.146054 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.146025    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:07:02.207401 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.207300    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T19:07:01.967810044Z","UUID":"9046613a-f9d7-451e-a78c-ee68e2757ca5","Handler":null,"Name":"","Endpoint":""}
Sep 17 19:07:02.208720 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.208702    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 19:07:02.208818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.208733    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 19:07:02.248938 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.248913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:02.249036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.248912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:02.249094 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:02.249035    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:02.249149 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:02.249112    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:02.249149 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.248912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:02.249249 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:02.249198    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:02.362891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.362847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"843296a8daa03642ea664e8cf68650527d022cc85b4a4423f255deddca6e6041"}
Sep 17 19:07:02.363353 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.362935    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"a3247d16f6b7bebffb5dd7dfce59d22c8cc156e5f0ebc78006da614f301cfa88"}
Sep 17 19:07:02.364998 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.364968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6" event={"ID":"9280330f-ef46-4bf3-84ed-7273c5f7ad58","Type":"ContainerStarted","Data":"9618d11c16cd3f40f8bd008846c9a5e0b00730a0bc069cbac132cde364ed4fed"}
Sep 17 19:07:02.366569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.366544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q5d75" event={"ID":"55b0adfc-769a-43b7-975d-e84c8603d651","Type":"ContainerStarted","Data":"f74b321d443c6e1178f741ff549eb99e1d0f043a22cef548f3a228da23cd295a"}
Sep 17 19:07:02.367587 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.367544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:07:02.368032 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.368013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4qvt9"
Sep 17 19:07:02.421116 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:02.421014    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q5d75" podStartSLOduration=4.782428078 podStartE2EDuration="22.420998255s" totalImagesPullingTime="17.638570177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:02.386604555 +0000 UTC m=+24.434484413" watchObservedRunningTime="2026-09-17 19:07:02.420998255 +0000 UTC m=+24.468878113"
Sep 17 19:07:03.371801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:03.371706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6" event={"ID":"9280330f-ef46-4bf3-84ed-7273c5f7ad58","Type":"ContainerStarted","Data":"0fcbda32dd2fbdbdeae470f982c2db2f5638a4d9b14e1276eb582e1f9d452927"}
Sep 17 19:07:03.393994 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:03.393936    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltxm6" podStartSLOduration=4.058613515 podStartE2EDuration="23.393918511s" totalImagesPullingTime="19.335304996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:03.393273571 +0000 UTC m=+25.441153420" watchObservedRunningTime="2026-09-17 19:07:03.393918511 +0000 UTC m=+25.441798368"
Sep 17 19:07:04.249742 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:04.249712    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:04.249742 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:04.249728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:04.249955 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:04.249847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:04.249955 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:04.249865    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:04.249955 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:04.249932    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:04.250123 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:04.250015    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:04.377568 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:04.377527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"ae568bb079b0ba61424ee9fd6e2e409edb7862f05dcea557fcbb953a79efa97c"}
Sep 17 19:07:06.249758 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.249530    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:06.250572 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.249889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:06.250572 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:06.250073    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:06.250572 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.250104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:06.250572 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:06.250485    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:06.250804 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:06.250758    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:06.383258 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.383233    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e76a145-f19d-4439-99c3-3501aae73b8d" containerID="7c0bc1bb7e797e76e3d0f1759521853c758609bbb44aa772e087f6d11f0146cf" exitCode=0
Sep 17 19:07:06.383383 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.383320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerDied","Data":"7c0bc1bb7e797e76e3d0f1759521853c758609bbb44aa772e087f6d11f0146cf"}
Sep 17 19:07:06.387295 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.387276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" event={"ID":"d7eebd66-0d16-4c8b-855b-c53da26cc8ec","Type":"ContainerStarted","Data":"52bcccd75e6df6daece9232a31dc56b9db578513697a4baa5e60ed0726e13c45"}
Sep 17 19:07:06.387579 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.387556    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:07:06.387662 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.387602    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:07:06.403305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.403285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:07:06.439985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:06.439938    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs" podStartSLOduration=8.768118059 podStartE2EDuration="26.439925545s" totalImagesPullingTime="17.671807486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:06.438098572 +0000 UTC m=+28.485978429" watchObservedRunningTime="2026-09-17 19:07:06.439925545 +0000 UTC m=+28.487805403"
Sep 17 19:07:07.389990 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:07.389819    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:07:07.404977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:07.404952    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:07:08.249524 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:08.249490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:08.249711 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:08.249503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:08.249711 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:08.249606    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:08.249711 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:08.249666    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:08.249711 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:08.249503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:08.249860 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:08.249760    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:08.394153 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:08.394124    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e76a145-f19d-4439-99c3-3501aae73b8d" containerID="bda5f7f688609e924f2f29d5a842860467fbf3e91977e82a63abd770cdf85d83" exitCode=0
Sep 17 19:07:08.394505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:08.394214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerDied","Data":"bda5f7f688609e924f2f29d5a842860467fbf3e91977e82a63abd770cdf85d83"}
Sep 17 19:07:10.252433 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:10.251397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:10.252433 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:10.251566    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:10.252433 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:10.251631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:10.252433 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:10.251748    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:10.252433 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:10.252196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:10.252433 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:10.252310    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:10.400531 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:10.400509    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e76a145-f19d-4439-99c3-3501aae73b8d" containerID="e1b5170c04513f1edf3b1a5f14c621b3daf46981e73fee52a32db8f17aa611c2" exitCode=0
Sep 17 19:07:10.400640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:10.400549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerDied","Data":"e1b5170c04513f1edf3b1a5f14c621b3daf46981e73fee52a32db8f17aa611c2"}
Sep 17 19:07:12.249667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:12.249625    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:12.250176 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:12.249636    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:12.250176 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:12.249785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:12.250176 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:12.249795    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:12.250176 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:12.249896    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:12.250176 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:12.249966    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:13.863331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:13.862996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:13.863880 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:13.863429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:13.863880 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.863461    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:07:13.863880 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.863561    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret podName:0d065b95-a94c-4a9f-a0b1-5f20f660c173 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:45.863532567 +0000 UTC m=+67.911412425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret") pod "global-pull-secret-syncer-tl6xh" (UID: "0d065b95-a94c-4a9f-a0b1-5f20f660c173") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:07:13.863880 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.863565    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:07:13.863880 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.863625    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs podName:35716ede-fc12-454a-a045-aca7fee96b1e nodeName:}" failed. No retries permitted until 2026-09-17 19:07:45.863608717 +0000 UTC m=+67.911488568 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs") pod "network-metrics-daemon-hpls8" (UID: "35716ede-fc12-454a-a045-aca7fee96b1e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:07:13.964573 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:13.964536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:13.964770 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.964745    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:07:13.964839 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.964774    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:07:13.964839 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.964789    2583 projected.go:196] Error preparing data for projected volume kube-api-access-5fsqw for pod openshift-network-diagnostics/network-check-target-d5lhg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:07:13.964914 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:13.964875    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw podName:7b583cdf-d259-4eae-b3fb-f929b34cd821 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:45.964856986 +0000 UTC m=+68.012736875 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5fsqw" (UniqueName: "kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw") pod "network-check-target-d5lhg" (UID: "7b583cdf-d259-4eae-b3fb-f929b34cd821") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:07:14.249885 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:14.249805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:14.250044 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:14.249804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:14.250044 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:14.249952    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:14.250044 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:14.249972    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:14.250180 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:14.250065    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:14.250180 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:14.250148    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:16.249300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:16.249264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:16.249300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:16.249282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:16.249753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:16.249269    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:16.249753 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:16.249366    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:16.249753 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:16.249430    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:16.249753 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:16.249544    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:16.417081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:16.417051    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e76a145-f19d-4439-99c3-3501aae73b8d" containerID="09c05a939e98c49c9ad93e3994459f78bbcd1db0cc787f0ff8097a98b1903459" exitCode=0
Sep 17 19:07:16.417229 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:16.417109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerDied","Data":"09c05a939e98c49c9ad93e3994459f78bbcd1db0cc787f0ff8097a98b1903459"}
Sep 17 19:07:17.422098 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:17.422067    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e76a145-f19d-4439-99c3-3501aae73b8d" containerID="c13cd9f03ae21141f83dbc5c9899c630bb5fec0def37ed91d8d6b55784013292" exitCode=0
Sep 17 19:07:17.422458 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:17.422130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerDied","Data":"c13cd9f03ae21141f83dbc5c9899c630bb5fec0def37ed91d8d6b55784013292"}
Sep 17 19:07:18.249119 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:18.249085    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:18.249119 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:18.249113    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:18.249326 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:18.249092    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:18.249326 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:18.249215    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:18.249326 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:18.249268    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:18.249423 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:18.249330    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:18.430598 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:18.430563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" event={"ID":"8e76a145-f19d-4439-99c3-3501aae73b8d","Type":"ContainerStarted","Data":"a5e7791a8d38628dea11cf3117e14ef19d675a923131c402854b3f13373b709d"}
Sep 17 19:07:18.478904 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:18.478850    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zz8lm" podStartSLOduration=9.378309405 podStartE2EDuration="38.47883251s" totalImagesPullingTime="29.100523105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:18.477427439 +0000 UTC m=+40.525307307" watchObservedRunningTime="2026-09-17 19:07:18.47883251 +0000 UTC m=+40.526712404"
Sep 17 19:07:19.819708 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:19.819430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tl6xh"]
Sep 17 19:07:19.820131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:19.819866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:19.820131 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:19.819964    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:19.825529 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:19.825499    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpls8"]
Sep 17 19:07:19.825709 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:19.825635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:19.825793 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:19.825769    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:19.828330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:19.828309    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d5lhg"]
Sep 17 19:07:19.828454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:19.828413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:19.828534 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:19.828513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:21.248772 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:21.248744    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:21.249290 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:21.248838    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:21.249290 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:21.248893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:21.249290 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:21.248977    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:22.249380 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:22.249349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:22.249748 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:22.249456    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpls8" podUID="35716ede-fc12-454a-a045-aca7fee96b1e"
Sep 17 19:07:23.249362 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.249325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:23.249523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.249327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:23.249523 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:23.249428    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d5lhg" podUID="7b583cdf-d259-4eae-b3fb-f929b34cd821"
Sep 17 19:07:23.249523 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:23.249501    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tl6xh" podUID="0d065b95-a94c-4a9f-a0b1-5f20f660c173"
Sep 17 19:07:23.834010 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.833979    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-131.ec2.internal" event="NodeReady"
Sep 17 19:07:23.834180 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.834088    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 19:07:23.901363 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.901333    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77bb5566d-pvt2v"]
Sep 17 19:07:23.903130 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.903109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:23.910394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.910348    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 19:07:23.910479 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.910438    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drwjl\""
Sep 17 19:07:23.910879 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.910862    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 19:07:23.911671 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.911657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 19:07:23.916979 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.916957    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 19:07:23.936004 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.935838    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pwqkr"]
Sep 17 19:07:23.937925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.937905    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bb5566d-pvt2v"]
Sep 17 19:07:23.938044 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.937940    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bmgtc"]
Sep 17 19:07:23.938103 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.938040    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:23.939733 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.939711    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:23.941056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.941036    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mr5fs\""
Sep 17 19:07:23.941265 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.941251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 19:07:23.941539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.941524    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 19:07:23.945732 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.945714    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 19:07:23.946262 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.946245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 19:07:23.947322 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.947304    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ftw2s\""
Sep 17 19:07:23.949160 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.949144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 19:07:23.949254 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.949146    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 19:07:23.958894 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.958873    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmgtc"]
Sep 17 19:07:23.964567 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:23.964545    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwqkr"]
Sep 17 19:07:24.028349 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.028312    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-42dpm"]
Sep 17 19:07:24.030168 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.030153    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.033513 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.033494    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 19:07:24.033728 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.033715    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4z8hc\""
Sep 17 19:07:24.033986 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.033971    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 19:07:24.034260 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034247    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 19:07:24.034525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5478\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-kube-api-access-h5478\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.034565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5052a006-9c61-40e2-be09-924837e29bb0-registry-certificates\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.034619 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5052a006-9c61-40e2-be09-924837e29bb0-trusted-ca\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.034667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5052a006-9c61-40e2-be09-924837e29bb0-installation-pull-secrets\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.034735 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-registry-tls\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.034800 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5052a006-9c61-40e2-be09-924837e29bb0-ca-trust-extracted\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.034855 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5052a006-9c61-40e2-be09-924837e29bb0-image-registry-private-configuration\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.034855 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.034841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-bound-sa-token\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.057294 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.057271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42dpm"]
Sep 17 19:07:24.135989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.135957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ln52\" (UniqueName: \"kubernetes.io/projected/822619f9-b256-49c2-be08-b881baa6346e-kube-api-access-8ln52\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.136146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqtc\" (UniqueName: \"kubernetes.io/projected/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-kube-api-access-lbqtc\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.136146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5478\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-kube-api-access-h5478\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.136146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136134    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-config-volume\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.136238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-tmp-dir\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.136238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/822619f9-b256-49c2-be08-b881baa6346e-crio-socket\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.136238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136181    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/822619f9-b256-49c2-be08-b881baa6346e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.136328 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b307565b-358c-426e-b984-e28122a91bb1-cert\") pod \"ingress-canary-42dpm\" (UID: \"b307565b-358c-426e-b984-e28122a91bb1\") " pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.136328 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-metrics-tls\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.136389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5052a006-9c61-40e2-be09-924837e29bb0-registry-certificates\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.136419 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5052a006-9c61-40e2-be09-924837e29bb0-trusted-ca\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.136451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/822619f9-b256-49c2-be08-b881baa6346e-data-volume\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.136488 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5052a006-9c61-40e2-be09-924837e29bb0-installation-pull-secrets\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.136532 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/822619f9-b256-49c2-be08-b881baa6346e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.136578 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xsg\" (UniqueName: \"kubernetes.io/projected/b307565b-358c-426e-b984-e28122a91bb1-kube-api-access-p6xsg\") pod \"ingress-canary-42dpm\" (UID: \"b307565b-358c-426e-b984-e28122a91bb1\") " pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.136621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-registry-tls\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.136660 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5052a006-9c61-40e2-be09-924837e29bb0-ca-trust-extracted\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.136747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5052a006-9c61-40e2-be09-924837e29bb0-image-registry-private-configuration\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.136810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.136754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-bound-sa-token\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.137038 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.137019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5052a006-9c61-40e2-be09-924837e29bb0-ca-trust-extracted\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.137335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.137315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5052a006-9c61-40e2-be09-924837e29bb0-registry-certificates\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.137498 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.137481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5052a006-9c61-40e2-be09-924837e29bb0-trusted-ca\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.140763 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.140739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5052a006-9c61-40e2-be09-924837e29bb0-installation-pull-secrets\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.140865 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.140794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-registry-tls\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.140865 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.140838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5052a006-9c61-40e2-be09-924837e29bb0-image-registry-private-configuration\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.156936 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.156912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-bound-sa-token\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.157935 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.157910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5478\" (UniqueName: \"kubernetes.io/projected/5052a006-9c61-40e2-be09-924837e29bb0-kube-api-access-h5478\") pod \"image-registry-77bb5566d-pvt2v\" (UID: \"5052a006-9c61-40e2-be09-924837e29bb0\") " pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.211972 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.211892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:24.237741 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.237715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ln52\" (UniqueName: \"kubernetes.io/projected/822619f9-b256-49c2-be08-b881baa6346e-kube-api-access-8ln52\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.237852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.237758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqtc\" (UniqueName: \"kubernetes.io/projected/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-kube-api-access-lbqtc\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.237852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.237800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-config-volume\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.237852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.237815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-tmp-dir\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.238025 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.237864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/822619f9-b256-49c2-be08-b881baa6346e-crio-socket\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.238025 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.237887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/822619f9-b256-49c2-be08-b881baa6346e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.238025 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.237911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b307565b-358c-426e-b984-e28122a91bb1-cert\") pod \"ingress-canary-42dpm\" (UID: \"b307565b-358c-426e-b984-e28122a91bb1\") " pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.238173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-metrics-tls\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.238173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/822619f9-b256-49c2-be08-b881baa6346e-data-volume\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.238173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/822619f9-b256-49c2-be08-b881baa6346e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.238307 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xsg\" (UniqueName: \"kubernetes.io/projected/b307565b-358c-426e-b984-e28122a91bb1-kube-api-access-p6xsg\") pod \"ingress-canary-42dpm\" (UID: \"b307565b-358c-426e-b984-e28122a91bb1\") " pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.238792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238438    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-tmp-dir\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.238792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/822619f9-b256-49c2-be08-b881baa6346e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.238792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-config-volume\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.238792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/822619f9-b256-49c2-be08-b881baa6346e-crio-socket\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.238792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.238752    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/822619f9-b256-49c2-be08-b881baa6346e-data-volume\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.240744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.240726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b307565b-358c-426e-b984-e28122a91bb1-cert\") pod \"ingress-canary-42dpm\" (UID: \"b307565b-358c-426e-b984-e28122a91bb1\") " pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.240835 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.240771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-metrics-tls\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.240909 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.240892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/822619f9-b256-49c2-be08-b881baa6346e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.249031 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.249007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:24.254854 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.254833    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:24.255733 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.255008    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4jlzv\""
Sep 17 19:07:24.255733 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.255041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ln52\" (UniqueName: \"kubernetes.io/projected/822619f9-b256-49c2-be08-b881baa6346e-kube-api-access-8ln52\") pod \"insights-runtime-extractor-bmgtc\" (UID: \"822619f9-b256-49c2-be08-b881baa6346e\") " pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.255733 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.255151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xsg\" (UniqueName: \"kubernetes.io/projected/b307565b-358c-426e-b984-e28122a91bb1-kube-api-access-p6xsg\") pod \"ingress-canary-42dpm\" (UID: \"b307565b-358c-426e-b984-e28122a91bb1\") " pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.255733 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.255527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqtc\" (UniqueName: \"kubernetes.io/projected/339bcb5f-ffaa-42f3-b10f-81dc7e391a5b-kube-api-access-lbqtc\") pod \"dns-default-pwqkr\" (UID: \"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b\") " pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.338108 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.338076    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42dpm"
Sep 17 19:07:24.545409 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.545169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bb5566d-pvt2v"]
Sep 17 19:07:24.547386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.547361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:24.548836 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:24.548807    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5052a006_9c61_40e2_be09_924837e29bb0.slice/crio-22b7197212753222caf796a288758dba5e09f211933d7d460f026821d6e906c3 WatchSource:0}: Error finding container 22b7197212753222caf796a288758dba5e09f211933d7d460f026821d6e906c3: Status 404 returned error can't find the container with id 22b7197212753222caf796a288758dba5e09f211933d7d460f026821d6e906c3
Sep 17 19:07:24.552226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.552201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmgtc"
Sep 17 19:07:24.632386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.632358    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42dpm"]
Sep 17 19:07:24.636199 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:24.636170    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb307565b_358c_426e_b984_e28122a91bb1.slice/crio-aa7733bd7ba2e472e8f4bdb9ec06d7a7b8affc95633b3b1fb248f64fbf22743d WatchSource:0}: Error finding container aa7733bd7ba2e472e8f4bdb9ec06d7a7b8affc95633b3b1fb248f64fbf22743d: Status 404 returned error can't find the container with id aa7733bd7ba2e472e8f4bdb9ec06d7a7b8affc95633b3b1fb248f64fbf22743d
Sep 17 19:07:24.833368 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.833329    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmgtc"]
Sep 17 19:07:24.836482 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:24.836451    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822619f9_b256_49c2_be08_b881baa6346e.slice/crio-188f44a730243339126413b5b3e240188f69aff9fe67378c6ecdf1878968e767 WatchSource:0}: Error finding container 188f44a730243339126413b5b3e240188f69aff9fe67378c6ecdf1878968e767: Status 404 returned error can't find the container with id 188f44a730243339126413b5b3e240188f69aff9fe67378c6ecdf1878968e767
Sep 17 19:07:24.849947 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:24.849923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pwqkr"]
Sep 17 19:07:24.853259 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:24.853235    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339bcb5f_ffaa_42f3_b10f_81dc7e391a5b.slice/crio-32e70feafd66cbb2e0a605f26559dba96b25be6a1f669169e477dfeca3d46835 WatchSource:0}: Error finding container 32e70feafd66cbb2e0a605f26559dba96b25be6a1f669169e477dfeca3d46835: Status 404 returned error can't find the container with id 32e70feafd66cbb2e0a605f26559dba96b25be6a1f669169e477dfeca3d46835
Sep 17 19:07:25.249525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.249445    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:25.249907 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.249890    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:25.252939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.252919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:07:25.253049 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.252954    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:07:25.254400 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.254377    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2b6xk\""
Sep 17 19:07:25.254498 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.254399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:25.448705 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.448658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42dpm" event={"ID":"b307565b-358c-426e-b984-e28122a91bb1","Type":"ContainerStarted","Data":"aa7733bd7ba2e472e8f4bdb9ec06d7a7b8affc95633b3b1fb248f64fbf22743d"}
Sep 17 19:07:25.450775 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.450738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bb5566d-pvt2v" event={"ID":"5052a006-9c61-40e2-be09-924837e29bb0","Type":"ContainerStarted","Data":"ed9fb734197b44951bb7491c0d968064d7045e4e6c92dae40342bb8b00ce8374"}
Sep 17 19:07:25.450909 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.450794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bb5566d-pvt2v" event={"ID":"5052a006-9c61-40e2-be09-924837e29bb0","Type":"ContainerStarted","Data":"22b7197212753222caf796a288758dba5e09f211933d7d460f026821d6e906c3"}
Sep 17 19:07:25.451654 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.451623    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:25.453652 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.453459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmgtc" event={"ID":"822619f9-b256-49c2-be08-b881baa6346e","Type":"ContainerStarted","Data":"9e9545af4719f2407a04b2b5d24d71448971959644d94697b528e787dc9351f3"}
Sep 17 19:07:25.453652 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.453512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmgtc" event={"ID":"822619f9-b256-49c2-be08-b881baa6346e","Type":"ContainerStarted","Data":"188f44a730243339126413b5b3e240188f69aff9fe67378c6ecdf1878968e767"}
Sep 17 19:07:25.455164 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.454953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwqkr" event={"ID":"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b","Type":"ContainerStarted","Data":"32e70feafd66cbb2e0a605f26559dba96b25be6a1f669169e477dfeca3d46835"}
Sep 17 19:07:25.481563 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:25.481205    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77bb5566d-pvt2v" podStartSLOduration=7.481179891 podStartE2EDuration="7.481179891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:25.480399914 +0000 UTC m=+47.528279773" watchObservedRunningTime="2026-09-17 19:07:25.481179891 +0000 UTC m=+47.529059749"
Sep 17 19:07:26.459280 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:26.459249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmgtc" event={"ID":"822619f9-b256-49c2-be08-b881baa6346e","Type":"ContainerStarted","Data":"8fde0ed13482905dcd071a409171ba1b3ad5cc797977370b081ff3e95c9d1db4"}
Sep 17 19:07:28.467189 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.466979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42dpm" event={"ID":"b307565b-358c-426e-b984-e28122a91bb1","Type":"ContainerStarted","Data":"b7e4151c7d460800a78e2271bdf11e34a60b36ae3058d4a47e85bb384c38da0f"}
Sep 17 19:07:28.469122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.469097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmgtc" event={"ID":"822619f9-b256-49c2-be08-b881baa6346e","Type":"ContainerStarted","Data":"e3abecd994b1d5b6022b6ed7664204bec3cae2bdb8af23af4ef1064cc424addd"}
Sep 17 19:07:28.471931 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.471900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwqkr" event={"ID":"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b","Type":"ContainerStarted","Data":"da4a9745236276622bb526743d0c5b0a7a57c8ae660590d5fbdf49eb73794290"}
Sep 17 19:07:28.472033 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.471944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pwqkr" event={"ID":"339bcb5f-ffaa-42f3-b10f-81dc7e391a5b","Type":"ContainerStarted","Data":"0d55865359e04de1ceffcf0ded336e2cb71ccd111670e172d3ec0821303a92db"}
Sep 17 19:07:28.472090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.472066    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:28.527791 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.527750    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-42dpm" podStartSLOduration=1.509792051 podStartE2EDuration="4.527734982s" totalImagesPullingTime="3.017942931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:28.488165291 +0000 UTC m=+50.536045147" watchObservedRunningTime="2026-09-17 19:07:28.527734982 +0000 UTC m=+50.575614817"
Sep 17 19:07:28.527995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.527973    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bmgtc" podStartSLOduration=2.824713306 podStartE2EDuration="5.527965655s" totalImagesPullingTime="2.703252349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:28.523289227 +0000 UTC m=+50.571169083" watchObservedRunningTime="2026-09-17 19:07:28.527965655 +0000 UTC m=+50.575845511"
Sep 17 19:07:28.545612 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:28.545568    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pwqkr" podStartSLOduration=2.759062829 podStartE2EDuration="5.545558781s" totalImagesPullingTime="2.786495952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:28.544841559 +0000 UTC m=+50.592721416" watchObservedRunningTime="2026-09-17 19:07:28.545558781 +0000 UTC m=+50.593438677"
Sep 17 19:07:34.509253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.509219    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"]
Sep 17 19:07:34.548425 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.548247    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pnkj5"]
Sep 17 19:07:34.548558 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.548361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.553852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.553834    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 19:07:34.554271 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.554255    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 19:07:34.555456 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.555435    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 19:07:34.555564 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.555454    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 19:07:34.555564 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.555500    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 19:07:34.555564 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.555435    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 19:07:34.555710 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.555454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r5r9l\""
Sep 17 19:07:34.562538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.562520    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"]
Sep 17 19:07:34.562629 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.562608    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.564964 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.564947    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 19:07:34.565584 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.565552    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wssgm\""
Sep 17 19:07:34.565584 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.565579    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 19:07:34.566771 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.566757    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 19:07:34.610050 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-sys\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-textfile\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-accelerators-collector-config\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.610142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.610268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-metrics-client-ca\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-wtmp\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wz7x\" (UniqueName: \"kubernetes.io/projected/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-kube-api-access-9wz7x\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.610388 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.610388 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-root\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610449 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-tls\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610449 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.610449 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.610535 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.610471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmmw\" (UniqueName: \"kubernetes.io/projected/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-api-access-kwmmw\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.711278 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.711403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-metrics-client-ca\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-wtmp\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wz7x\" (UniqueName: \"kubernetes.io/projected/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-kube-api-access-9wz7x\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.711604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.711604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-root\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-wtmp\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-root\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-tls\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.711945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711676    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmmw\" (UniqueName: \"kubernetes.io/projected/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-api-access-kwmmw\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.711945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-sys\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-textfile\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-accelerators-collector-config\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-sys\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.711945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.712300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.711979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-metrics-client-ca\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.712300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.712151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-textfile\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.712300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.712171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.712467 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:34.712305    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 19:07:34.712467 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.712452    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.712550 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.712463    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.712550 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:34.712539    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-tls podName:aeeef8c0-6a47-4686-bfae-6a0b500d09e0 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:35.212520613 +0000 UTC m=+57.260400462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-jlb8k" (UID: "aeeef8c0-6a47-4686-bfae-6a0b500d09e0") : secret "kube-state-metrics-tls" not found
Sep 17 19:07:34.712758 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.712733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-accelerators-collector-config\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.715631 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.715604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-tls\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.715731 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.715651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.715795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.715777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.726783 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.726757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wz7x\" (UniqueName: \"kubernetes.io/projected/9bf21b45-0fc6-4ff3-8a74-d5b9b0423522-kube-api-access-9wz7x\") pod \"node-exporter-pnkj5\" (UID: \"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522\") " pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:34.727161 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.727139    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmmw\" (UniqueName: \"kubernetes.io/projected/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-api-access-kwmmw\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:34.870655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:34.870625    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pnkj5"
Sep 17 19:07:35.216188 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.216131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:35.218546 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.218526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeeef8c0-6a47-4686-bfae-6a0b500d09e0-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-jlb8k\" (UID: \"aeeef8c0-6a47-4686-bfae-6a0b500d09e0\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:35.452884 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.452850    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:35.457389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.457369    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"
Sep 17 19:07:35.476837 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.476762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.479772 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.479752    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.480716    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.480768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.480792    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jwl5c\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.480851    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.480716    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.480928    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.481096    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.481165    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.481174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 19:07:35.481296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.480791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 19:07:35.494818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.494795    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pnkj5" event={"ID":"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522","Type":"ContainerStarted","Data":"11b0da36bbac7243e48c38c6b73f4b2f60e43623905bd2efee94f63fee11e10a"}
Sep 17 19:07:35.518085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jvw\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-kube-api-access-25jvw\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-out\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-volume\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-web-config\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.518525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.518458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.619892 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.619852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620050 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.619911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620050 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.619934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620050 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jvw\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-kube-api-access-25jvw\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-out\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-volume\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.620330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620311    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-web-config\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.621023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.620988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.621316 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:35.621247    2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 19:07:35.621384 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:07:35.621322    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls podName:d583f55a-d7ea-4ade-9b4d-3be07a8da97a nodeName:}" failed. No retries permitted until 2026-09-17 19:07:36.121303016 +0000 UTC m=+58.169182857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a") : secret "alertmanager-main-tls" not found
Sep 17 19:07:35.623090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.622484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.623090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.622621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.623247 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.623199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-web-config\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.624081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.624041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.624538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.624518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-volume\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.625147 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.625122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.625550 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.625530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.625633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.625567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.625755 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.625734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.626212 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.626192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-out\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.632975 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.632957    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jvw\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-kube-api-access-25jvw\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:35.760370 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:35.760346    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-jlb8k"]
Sep 17 19:07:35.979180 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:35.979146    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeeef8c0_6a47_4686_bfae_6a0b500d09e0.slice/crio-c8a895d421eedc2a1cc4ad1fa9d7897aa9e2272b0d40769dd05265bbb17a0444 WatchSource:0}: Error finding container c8a895d421eedc2a1cc4ad1fa9d7897aa9e2272b0d40769dd05265bbb17a0444: Status 404 returned error can't find the container with id c8a895d421eedc2a1cc4ad1fa9d7897aa9e2272b0d40769dd05265bbb17a0444
Sep 17 19:07:36.125789 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.125756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:36.128171 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.128149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:36.389797 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.389776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:36.457944 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.457914    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"]
Sep 17 19:07:36.470782 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.470757    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.474821 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.474793    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 19:07:36.475765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.475016    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 19:07:36.475765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.475277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"]
Sep 17 19:07:36.475765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.475390    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 19:07:36.475765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.475500    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-av67v4oh5ms0f\""
Sep 17 19:07:36.475765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.475557    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 19:07:36.475765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.475751    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 19:07:36.476121 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.475929    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2c89s\""
Sep 17 19:07:36.499475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.499445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k" event={"ID":"aeeef8c0-6a47-4686-bfae-6a0b500d09e0","Type":"ContainerStarted","Data":"c8a895d421eedc2a1cc4ad1fa9d7897aa9e2272b0d40769dd05265bbb17a0444"}
Sep 17 19:07:36.501073 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.501042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pnkj5" event={"ID":"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522","Type":"ContainerStarted","Data":"01971be055b3b9717b26bd401897a342f8b0331866474e40cf40b0fa6ba527f8"}
Sep 17 19:07:36.530049 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.530317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-grpc-tls\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.530317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530102    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.530317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.530317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530282    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d120838-aa40-46bb-bc83-cc659f36da0c-metrics-client-ca\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.530525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsm6r\" (UniqueName: \"kubernetes.io/projected/4d120838-aa40-46bb-bc83-cc659f36da0c-kube-api-access-qsm6r\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.530525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.530525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.530432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-tls\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.631741 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.631711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d120838-aa40-46bb-bc83-cc659f36da0c-metrics-client-ca\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.631879 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.631761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsm6r\" (UniqueName: \"kubernetes.io/projected/4d120838-aa40-46bb-bc83-cc659f36da0c-kube-api-access-qsm6r\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.631879 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.631818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.631879 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.631868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-tls\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.632016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.631890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.632016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.631913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-grpc-tls\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.632016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.631938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.632157 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.632126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.632588 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.632514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d120838-aa40-46bb-bc83-cc659f36da0c-metrics-client-ca\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.635114 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.635088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.635264 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.635149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-tls\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.635264 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.635222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.635387 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.635309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.635569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.635551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.635811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.635791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d120838-aa40-46bb-bc83-cc659f36da0c-secret-grpc-tls\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.641016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.640969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsm6r\" (UniqueName: \"kubernetes.io/projected/4d120838-aa40-46bb-bc83-cc659f36da0c-kube-api-access-qsm6r\") pod \"thanos-querier-67bb8c8f5c-sbrt6\" (UID: \"4d120838-aa40-46bb-bc83-cc659f36da0c\") " pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:36.697360 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.697336    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:36.699332 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:36.699311    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd583f55a_d7ea_4ade_9b4d_3be07a8da97a.slice/crio-0d2674f430df7484a23fd2273e4d3f48f83edfc21444021cdbac434500fa9482 WatchSource:0}: Error finding container 0d2674f430df7484a23fd2273e4d3f48f83edfc21444021cdbac434500fa9482: Status 404 returned error can't find the container with id 0d2674f430df7484a23fd2273e4d3f48f83edfc21444021cdbac434500fa9482
Sep 17 19:07:36.784431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:36.784407    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:37.100527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:37.100490    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"]
Sep 17 19:07:37.349907 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:37.349876    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d120838_aa40_46bb_bc83_cc659f36da0c.slice/crio-c6253e668c98009608b3b1eca3d9eebeecf1d3abcd0a1b1594383bae8c07a6d3 WatchSource:0}: Error finding container c6253e668c98009608b3b1eca3d9eebeecf1d3abcd0a1b1594383bae8c07a6d3: Status 404 returned error can't find the container with id c6253e668c98009608b3b1eca3d9eebeecf1d3abcd0a1b1594383bae8c07a6d3
Sep 17 19:07:37.505583 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:37.505555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerStarted","Data":"0d2674f430df7484a23fd2273e4d3f48f83edfc21444021cdbac434500fa9482"}
Sep 17 19:07:37.507134 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:37.507102    2583 generic.go:356] "Generic (PLEG): container finished" podID="9bf21b45-0fc6-4ff3-8a74-d5b9b0423522" containerID="01971be055b3b9717b26bd401897a342f8b0331866474e40cf40b0fa6ba527f8" exitCode=0
Sep 17 19:07:37.507231 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:37.507191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pnkj5" event={"ID":"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522","Type":"ContainerDied","Data":"01971be055b3b9717b26bd401897a342f8b0331866474e40cf40b0fa6ba527f8"}
Sep 17 19:07:37.508447 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:37.508426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" event={"ID":"4d120838-aa40-46bb-bc83-cc659f36da0c","Type":"ContainerStarted","Data":"c6253e668c98009608b3b1eca3d9eebeecf1d3abcd0a1b1594383bae8c07a6d3"}
Sep 17 19:07:38.477560 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.477532    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pwqkr"
Sep 17 19:07:38.515154 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.515125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pnkj5" event={"ID":"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522","Type":"ContainerStarted","Data":"d5b171f42455ea9d7a19a1a3084289fba2dca4bccbd7e4cd8dfe03a567a0e59e"}
Sep 17 19:07:38.515298 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.515179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pnkj5" event={"ID":"9bf21b45-0fc6-4ff3-8a74-d5b9b0423522","Type":"ContainerStarted","Data":"19164fb545ccebea1237313f3015fd7890c0532571e96d84b302a9790e63d701"}
Sep 17 19:07:38.519125 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.519096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k" event={"ID":"aeeef8c0-6a47-4686-bfae-6a0b500d09e0","Type":"ContainerStarted","Data":"2d3c9b457a39efcb8a86767752b933bf99c0b654306472941e4fa52470755573"}
Sep 17 19:07:38.519245 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.519154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k" event={"ID":"aeeef8c0-6a47-4686-bfae-6a0b500d09e0","Type":"ContainerStarted","Data":"666c595d5731950c673bd47a59bc74fa14626ff23feb749e57104d51d51055e3"}
Sep 17 19:07:38.519245 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.519182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k" event={"ID":"aeeef8c0-6a47-4686-bfae-6a0b500d09e0","Type":"ContainerStarted","Data":"ddd3759effcfa0334d9997241abba1ea4df7d0447f3665cae132dd0c03b26a96"}
Sep 17 19:07:38.540468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.540420    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pnkj5" podStartSLOduration=3.03604767 podStartE2EDuration="4.540401841s" totalImagesPullingTime="1.414272027s" totalInitContainerRuntime="90.082144ms" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:38.539870089 +0000 UTC m=+60.587749947" watchObservedRunningTime="2026-09-17 19:07:38.540401841 +0000 UTC m=+60.588281699"
Sep 17 19:07:38.560527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:38.560484    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-jlb8k" podStartSLOduration=2.826735084 podStartE2EDuration="4.560469037s" totalImagesPullingTime="1.733733953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:38.559142385 +0000 UTC m=+60.607022242" watchObservedRunningTime="2026-09-17 19:07:38.560469037 +0000 UTC m=+60.608348894"
Sep 17 19:07:39.411041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:39.411014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgdhs"
Sep 17 19:07:39.524020 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:39.523992    2583 generic.go:356] "Generic (PLEG): container finished" podID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerID="344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d" exitCode=0
Sep 17 19:07:39.524374 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:39.524080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d"}
Sep 17 19:07:40.529871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:40.529832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" event={"ID":"4d120838-aa40-46bb-bc83-cc659f36da0c","Type":"ContainerStarted","Data":"8944ca3d1e7d91ab88e8a34bee7dbeab489c3dc54c069509a8a723e7b529ac9f"}
Sep 17 19:07:40.530281 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:40.529898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" event={"ID":"4d120838-aa40-46bb-bc83-cc659f36da0c","Type":"ContainerStarted","Data":"3e5bf8f40e56fab8db13dc94eb56314c66b61883dcd23fd78389a6740948236e"}
Sep 17 19:07:40.530281 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:40.529924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" event={"ID":"4d120838-aa40-46bb-bc83-cc659f36da0c","Type":"ContainerStarted","Data":"1c35fc7071ff68dcbf04aafac42f51272ea42cc6c3f98a868a3a3f1ff4bcb2a2"}
Sep 17 19:07:42.539441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.539350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" event={"ID":"4d120838-aa40-46bb-bc83-cc659f36da0c","Type":"ContainerStarted","Data":"9478e9199ca8f9cf3ffde8f47a4549eafae7a59d1132554839e464038b40a7ab"}
Sep 17 19:07:42.539441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.539435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" event={"ID":"4d120838-aa40-46bb-bc83-cc659f36da0c","Type":"ContainerStarted","Data":"554e345255cf85a6009c8e84409a0003f5db34ebec77067d5d72f62839776442"}
Sep 17 19:07:42.539939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.539489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:42.539939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.539531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" event={"ID":"4d120838-aa40-46bb-bc83-cc659f36da0c","Type":"ContainerStarted","Data":"83ecd8bb799f9021b582ce22dfa86d749073227b9764eabf919540cda9e22909"}
Sep 17 19:07:42.543526 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.543493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerStarted","Data":"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"}
Sep 17 19:07:42.543641 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.543570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerStarted","Data":"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"}
Sep 17 19:07:42.543641 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.543613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerStarted","Data":"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"}
Sep 17 19:07:42.543783 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.543654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerStarted","Data":"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"}
Sep 17 19:07:42.543783 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.543726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerStarted","Data":"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"}
Sep 17 19:07:42.543783 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.543754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerStarted","Data":"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"}
Sep 17 19:07:42.568999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.568948    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6" podStartSLOduration=2.618101986 podStartE2EDuration="6.568934s" totalImagesPullingTime="3.950832014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:42.567865606 +0000 UTC m=+64.615745462" watchObservedRunningTime="2026-09-17 19:07:42.568934 +0000 UTC m=+64.616813858"
Sep 17 19:07:42.606621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:42.606576    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.506042579 podStartE2EDuration="7.606558083s" totalImagesPullingTime="4.079565089s" totalInitContainerRuntime="20.950415ms" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:42.60387017 +0000 UTC m=+64.651750042" watchObservedRunningTime="2026-09-17 19:07:42.606558083 +0000 UTC m=+64.654437941"
Sep 17 19:07:43.075113 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.075079    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f957498fb-6ht5h"]
Sep 17 19:07:43.077491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.077472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.080194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.080159    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7lrc9\""
Sep 17 19:07:43.080492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.080473    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 19:07:43.080767 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.080742    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 19:07:43.080884 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.080773    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 19:07:43.081385 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.081365    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 19:07:43.081385 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.081383    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 19:07:43.081522 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.081426    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 19:07:43.081522 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.081430    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 19:07:43.095085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.095054    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f957498fb-6ht5h"]
Sep 17 19:07:43.191238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.191210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-oauth-config\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.191238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.191239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hn7\" (UniqueName: \"kubernetes.io/projected/a691241d-1495-480a-a02d-3df4bc18299d-kube-api-access-d7hn7\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.191410 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.191258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-service-ca\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.191410 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.191279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-console-config\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.191410 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.191344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-oauth-serving-cert\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.191559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.191489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-serving-cert\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.292292 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.292270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-oauth-config\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.292386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.292297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hn7\" (UniqueName: \"kubernetes.io/projected/a691241d-1495-480a-a02d-3df4bc18299d-kube-api-access-d7hn7\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.292386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.292316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-service-ca\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.292484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.292453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-console-config\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.292535 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.292495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-oauth-serving-cert\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.292622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.292596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-serving-cert\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.293103 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.293069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-service-ca\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.293215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.293195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-oauth-serving-cert\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.293278 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.293221    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-console-config\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.294919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.294895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-oauth-config\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.295138 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.295118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-serving-cert\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.301290 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.301271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hn7\" (UniqueName: \"kubernetes.io/projected/a691241d-1495-480a-a02d-3df4bc18299d-kube-api-access-d7hn7\") pod \"console-5f957498fb-6ht5h\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") " pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.387910 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.387857    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:43.661446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:43.661357    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f957498fb-6ht5h"]
Sep 17 19:07:43.663618 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:43.663592    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda691241d_1495_480a_a02d_3df4bc18299d.slice/crio-c0046a1d3c51442d62725665c5c7fa682e7858b91f8c756d67afa5c997ba063c WatchSource:0}: Error finding container c0046a1d3c51442d62725665c5c7fa682e7858b91f8c756d67afa5c997ba063c: Status 404 returned error can't find the container with id c0046a1d3c51442d62725665c5c7fa682e7858b91f8c756d67afa5c997ba063c
Sep 17 19:07:44.554163 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:44.554085    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f957498fb-6ht5h" event={"ID":"a691241d-1495-480a-a02d-3df4bc18299d","Type":"ContainerStarted","Data":"c0046a1d3c51442d62725665c5c7fa682e7858b91f8c756d67afa5c997ba063c"}
Sep 17 19:07:45.916750 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:45.916714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:45.917143 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:45.916834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:45.920115 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:45.920091    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:45.921208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:45.921194    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:45.930249 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:45.930224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35716ede-fc12-454a-a045-aca7fee96b1e-metrics-certs\") pod \"network-metrics-daemon-hpls8\" (UID: \"35716ede-fc12-454a-a045-aca7fee96b1e\") " pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:45.930249 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:45.930236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d065b95-a94c-4a9f-a0b1-5f20f660c173-original-pull-secret\") pod \"global-pull-secret-syncer-tl6xh\" (UID: \"0d065b95-a94c-4a9f-a0b1-5f20f660c173\") " pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:45.969501 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:45.969475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tl6xh"
Sep 17 19:07:46.018082 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.018049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:46.020886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.020864    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:07:46.031129 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.031104    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:07:46.041275 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.041253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsqw\" (UniqueName: \"kubernetes.io/projected/7b583cdf-d259-4eae-b3fb-f929b34cd821-kube-api-access-5fsqw\") pod \"network-check-target-d5lhg\" (UID: \"7b583cdf-d259-4eae-b3fb-f929b34cd821\") " pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:46.165526 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.165499    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4jlzv\""
Sep 17 19:07:46.171228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.171165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpls8"
Sep 17 19:07:46.265848 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.265817    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2b6xk\""
Sep 17 19:07:46.273604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:46.273585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:47.468072 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:47.468047    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77bb5566d-pvt2v"
Sep 17 19:07:47.812871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:47.812733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tl6xh"]
Sep 17 19:07:47.815316 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:47.815280    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d065b95_a94c_4a9f_a0b1_5f20f660c173.slice/crio-8972a98aad32d9c81edd8b9f88c254e7258279375c7fe39d927be4cfc8f8c7af WatchSource:0}: Error finding container 8972a98aad32d9c81edd8b9f88c254e7258279375c7fe39d927be4cfc8f8c7af: Status 404 returned error can't find the container with id 8972a98aad32d9c81edd8b9f88c254e7258279375c7fe39d927be4cfc8f8c7af
Sep 17 19:07:47.818424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:47.818387    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d5lhg"]
Sep 17 19:07:47.821356 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:47.821336    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b583cdf_d259_4eae_b3fb_f929b34cd821.slice/crio-b3ae6561f52bae6277888bc80096093edf40542f0f817db007bf13454ec9ffc5 WatchSource:0}: Error finding container b3ae6561f52bae6277888bc80096093edf40542f0f817db007bf13454ec9ffc5: Status 404 returned error can't find the container with id b3ae6561f52bae6277888bc80096093edf40542f0f817db007bf13454ec9ffc5
Sep 17 19:07:47.841128 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:47.841108    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpls8"]
Sep 17 19:07:47.843760 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:47.843737    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35716ede_fc12_454a_a045_aca7fee96b1e.slice/crio-6c6b35e262b25f644bcd0bf518793051279e295d51a03fde7dd1cce870aa21d4 WatchSource:0}: Error finding container 6c6b35e262b25f644bcd0bf518793051279e295d51a03fde7dd1cce870aa21d4: Status 404 returned error can't find the container with id 6c6b35e262b25f644bcd0bf518793051279e295d51a03fde7dd1cce870aa21d4
Sep 17 19:07:48.555867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:48.555834    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67bb8c8f5c-sbrt6"
Sep 17 19:07:48.575667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:48.575617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tl6xh" event={"ID":"0d065b95-a94c-4a9f-a0b1-5f20f660c173","Type":"ContainerStarted","Data":"8972a98aad32d9c81edd8b9f88c254e7258279375c7fe39d927be4cfc8f8c7af"}
Sep 17 19:07:48.578015 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:48.577987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f957498fb-6ht5h" event={"ID":"a691241d-1495-480a-a02d-3df4bc18299d","Type":"ContainerStarted","Data":"1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8"}
Sep 17 19:07:48.579853 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:48.579827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpls8" event={"ID":"35716ede-fc12-454a-a045-aca7fee96b1e","Type":"ContainerStarted","Data":"6c6b35e262b25f644bcd0bf518793051279e295d51a03fde7dd1cce870aa21d4"}
Sep 17 19:07:48.581327 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:48.581300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d5lhg" event={"ID":"7b583cdf-d259-4eae-b3fb-f929b34cd821","Type":"ContainerStarted","Data":"b3ae6561f52bae6277888bc80096093edf40542f0f817db007bf13454ec9ffc5"}
Sep 17 19:07:48.608392 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:48.608343    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f957498fb-6ht5h" podStartSLOduration=1.69383557 podStartE2EDuration="5.608328807s" totalImagesPullingTime="3.914493237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:48.607524386 +0000 UTC m=+70.655404244" watchObservedRunningTime="2026-09-17 19:07:48.608328807 +0000 UTC m=+70.656208663"
Sep 17 19:07:49.588799 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:49.588504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpls8" event={"ID":"35716ede-fc12-454a-a045-aca7fee96b1e","Type":"ContainerStarted","Data":"f545b249c1cd69d7f86d0cc5ab375f550194ce0fc5eddb464c1fd084765a54ef"}
Sep 17 19:07:49.588799 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:49.588555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpls8" event={"ID":"35716ede-fc12-454a-a045-aca7fee96b1e","Type":"ContainerStarted","Data":"2a2e4fefce73abdd185b3a860c8ae1abe1a7643e887b04cea0eaea7f6a00d624"}
Sep 17 19:07:49.612585 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:49.612460    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hpls8" podStartSLOduration=68.435891677 podStartE2EDuration="1m9.612441676s" totalImagesPullingTime="1.176549999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:49.611564106 +0000 UTC m=+71.659443974" watchObservedRunningTime="2026-09-17 19:07:49.612441676 +0000 UTC m=+71.660321559"
Sep 17 19:07:52.604072 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.604033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d5lhg" event={"ID":"7b583cdf-d259-4eae-b3fb-f929b34cd821","Type":"ContainerStarted","Data":"b4b5f60493b7d2e73c11bc2e5f2f4810b25e738994f04810115235d318a3fa5f"}
Sep 17 19:07:52.604478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.604139    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:07:52.605756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.605731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tl6xh" event={"ID":"0d065b95-a94c-4a9f-a0b1-5f20f660c173","Type":"ContainerStarted","Data":"59d3d7cc35af81296fe30d7c9f1bca64498cf60aa81bb971d563ea12f1506876"}
Sep 17 19:07:52.632039 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.631992    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d5lhg" podStartSLOduration=68.198192635 podStartE2EDuration="1m12.63197875s" totalImagesPullingTime="4.433786115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:52.630253233 +0000 UTC m=+74.678133090" watchObservedRunningTime="2026-09-17 19:07:52.63197875 +0000 UTC m=+74.679858646"
Sep 17 19:07:52.669286 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.669248    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tl6xh" podStartSLOduration=68.22541797 podStartE2EDuration="1m12.669236985s" totalImagesPullingTime="4.443819015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:52.666786842 +0000 UTC m=+74.714666710" watchObservedRunningTime="2026-09-17 19:07:52.669236985 +0000 UTC m=+74.717116838"
Sep 17 19:07:52.963940 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.963873    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f97889b4-ljpj2"]
Sep 17 19:07:52.966043 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.966026    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:52.978584 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.978559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 19:07:52.992240 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:52.992218    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f97889b4-ljpj2"]
Sep 17 19:07:53.077028 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.077004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-trusted-ca-bundle\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.077150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.077047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-service-ca\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.077150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.077075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-oauth-config\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.077150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.077103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snk4\" (UniqueName: \"kubernetes.io/projected/04bf16b4-87b1-4dd3-8f11-104d036a16cf-kube-api-access-4snk4\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.077150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.077141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-serving-cert\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.077277 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.077168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-config\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.077277 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.077197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-oauth-serving-cert\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.177635 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.177606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-service-ca\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.177782 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.177654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-oauth-config\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.177782 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.177710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snk4\" (UniqueName: \"kubernetes.io/projected/04bf16b4-87b1-4dd3-8f11-104d036a16cf-kube-api-access-4snk4\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.177888 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.177797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-serving-cert\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.177888 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.177849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-config\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.177888 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.177874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-oauth-serving-cert\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.178082 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.178052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-trusted-ca-bundle\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.178397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.178363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-service-ca\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.178577 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.178557    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-config\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.178657 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.178611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-oauth-serving-cert\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.179222 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.179197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-trusted-ca-bundle\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.180780 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.180751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-serving-cert\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.180828 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.180751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-oauth-config\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.188859 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.188835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snk4\" (UniqueName: \"kubernetes.io/projected/04bf16b4-87b1-4dd3-8f11-104d036a16cf-kube-api-access-4snk4\") pod \"console-79f97889b4-ljpj2\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") " pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.275793 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.275742    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:07:53.388459 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.388428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:53.388600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.388482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:53.392961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.392941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:53.553114 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.553086    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f97889b4-ljpj2"]
Sep 17 19:07:53.555620 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:07:53.555596    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bf16b4_87b1_4dd3_8f11_104d036a16cf.slice/crio-bfd9c3d339cb5212c681217bfe0355c15707b883939a605f8ef94a82c37d07f6 WatchSource:0}: Error finding container bfd9c3d339cb5212c681217bfe0355c15707b883939a605f8ef94a82c37d07f6: Status 404 returned error can't find the container with id bfd9c3d339cb5212c681217bfe0355c15707b883939a605f8ef94a82c37d07f6
Sep 17 19:07:53.611213 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.611185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f97889b4-ljpj2" event={"ID":"04bf16b4-87b1-4dd3-8f11-104d036a16cf","Type":"ContainerStarted","Data":"bfd9c3d339cb5212c681217bfe0355c15707b883939a605f8ef94a82c37d07f6"}
Sep 17 19:07:53.615054 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:53.615036    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:07:54.616247 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:54.616198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f97889b4-ljpj2" event={"ID":"04bf16b4-87b1-4dd3-8f11-104d036a16cf","Type":"ContainerStarted","Data":"4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f"}
Sep 17 19:07:54.643144 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:07:54.643089    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79f97889b4-ljpj2" podStartSLOduration=2.643073747 podStartE2EDuration="2.643073747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:54.642463049 +0000 UTC m=+76.690342907" watchObservedRunningTime="2026-09-17 19:07:54.643073747 +0000 UTC m=+76.690953603"
Sep 17 19:08:03.276803 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:03.276759    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:08:03.276803 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:03.276812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:08:03.281672 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:03.281647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:08:03.657170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:03.657133    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:08:03.713044 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:03.713005    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f957498fb-6ht5h"]
Sep 17 19:08:23.613331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:23.613291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d5lhg"
Sep 17 19:08:28.734449 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:28.734401    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f957498fb-6ht5h" podUID="a691241d-1495-480a-a02d-3df4bc18299d" containerName="console" containerID="cri-o://1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8" gracePeriod=15
Sep 17 19:08:28.984106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:28.984076    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:08:29.150450 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150422    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-console-config\") pod \"a691241d-1495-480a-a02d-3df4bc18299d\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") "
Sep 17 19:08:29.150582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-serving-cert\") pod \"a691241d-1495-480a-a02d-3df4bc18299d\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") "
Sep 17 19:08:29.150582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-oauth-serving-cert\") pod \"a691241d-1495-480a-a02d-3df4bc18299d\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") "
Sep 17 19:08:29.150582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150529    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7hn7\" (UniqueName: \"kubernetes.io/projected/a691241d-1495-480a-a02d-3df4bc18299d-kube-api-access-d7hn7\") pod \"a691241d-1495-480a-a02d-3df4bc18299d\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") "
Sep 17 19:08:29.150582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150552    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-oauth-config\") pod \"a691241d-1495-480a-a02d-3df4bc18299d\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") "
Sep 17 19:08:29.150582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150572    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-service-ca\") pod \"a691241d-1495-480a-a02d-3df4bc18299d\" (UID: \"a691241d-1495-480a-a02d-3df4bc18299d\") "
Sep 17 19:08:29.150948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150853    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-console-config" (OuterVolumeSpecName: "console-config") pod "a691241d-1495-480a-a02d-3df4bc18299d" (UID: "a691241d-1495-480a-a02d-3df4bc18299d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:29.151057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.150952    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a691241d-1495-480a-a02d-3df4bc18299d" (UID: "a691241d-1495-480a-a02d-3df4bc18299d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:29.151057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.151008    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-service-ca" (OuterVolumeSpecName: "service-ca") pod "a691241d-1495-480a-a02d-3df4bc18299d" (UID: "a691241d-1495-480a-a02d-3df4bc18299d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:29.152945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.152918    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a691241d-1495-480a-a02d-3df4bc18299d" (UID: "a691241d-1495-480a-a02d-3df4bc18299d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:29.153041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.152969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a691241d-1495-480a-a02d-3df4bc18299d-kube-api-access-d7hn7" (OuterVolumeSpecName: "kube-api-access-d7hn7") pod "a691241d-1495-480a-a02d-3df4bc18299d" (UID: "a691241d-1495-480a-a02d-3df4bc18299d"). InnerVolumeSpecName "kube-api-access-d7hn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:29.153041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.152970    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a691241d-1495-480a-a02d-3df4bc18299d" (UID: "a691241d-1495-480a-a02d-3df4bc18299d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:29.251553 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.251521    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hn7\" (UniqueName: \"kubernetes.io/projected/a691241d-1495-480a-a02d-3df4bc18299d-kube-api-access-d7hn7\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:29.251553 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.251546    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-oauth-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:29.251553 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.251556    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:29.251553 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.251565    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-console-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:29.251816 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.251573    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a691241d-1495-480a-a02d-3df4bc18299d-console-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:29.251816 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.251581    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a691241d-1495-480a-a02d-3df4bc18299d-oauth-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:29.758380 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.758343    2583 generic.go:356] "Generic (PLEG): container finished" podID="a691241d-1495-480a-a02d-3df4bc18299d" containerID="1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8" exitCode=0
Sep 17 19:08:29.758831 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.758408    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f957498fb-6ht5h"
Sep 17 19:08:29.758831 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.758457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f957498fb-6ht5h" event={"ID":"a691241d-1495-480a-a02d-3df4bc18299d","Type":"ContainerDied","Data":"1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8"}
Sep 17 19:08:29.758831 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.758522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f957498fb-6ht5h" event={"ID":"a691241d-1495-480a-a02d-3df4bc18299d","Type":"ContainerDied","Data":"c0046a1d3c51442d62725665c5c7fa682e7858b91f8c756d67afa5c997ba063c"}
Sep 17 19:08:29.758831 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.758551    2583 scope.go:117] "RemoveContainer" containerID="1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8"
Sep 17 19:08:29.768455 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.768436    2583 scope.go:117] "RemoveContainer" containerID="1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8"
Sep 17 19:08:29.768793 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:29.768765    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8\": container with ID starting with 1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8 not found: ID does not exist" containerID="1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8"
Sep 17 19:08:29.768908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.768799    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8"} err="failed to get container status \"1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8\": rpc error: code = NotFound desc = could not find container \"1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8\": container with ID starting with 1e6488c86b726ff9565d9e547f480af9070b34e97c3f44038a40ebd7cf1188f8 not found: ID does not exist"
Sep 17 19:08:29.789128 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.789106    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f957498fb-6ht5h"]
Sep 17 19:08:29.791607 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:29.791579    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f957498fb-6ht5h"]
Sep 17 19:08:30.253718 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:30.253669    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a691241d-1495-480a-a02d-3df4bc18299d" path="/var/lib/kubelet/pods/a691241d-1495-480a-a02d-3df4bc18299d/volumes"
Sep 17 19:08:54.764907 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:54.764873    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:08:54.765331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:54.765186    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="alertmanager" containerID="cri-o://a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd" gracePeriod=120
Sep 17 19:08:54.765331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:54.765262    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-metric" containerID="cri-o://d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617" gracePeriod=120
Sep 17 19:08:54.765462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:54.765319    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="config-reloader" containerID="cri-o://0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e" gracePeriod=120
Sep 17 19:08:54.765462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:54.765289    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-web" containerID="cri-o://9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac" gracePeriod=120
Sep 17 19:08:54.765462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:54.765368    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="prom-label-proxy" containerID="cri-o://5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a" gracePeriod=120
Sep 17 19:08:54.765462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:54.765319    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy" containerID="cri-o://44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2" gracePeriod=120
Sep 17 19:08:55.863114 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863088    2583 generic.go:356] "Generic (PLEG): container finished" podID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerID="5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a" exitCode=0
Sep 17 19:08:55.863114 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863112    2583 generic.go:356] "Generic (PLEG): container finished" podID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerID="44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2" exitCode=0
Sep 17 19:08:55.863500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863121    2583 generic.go:356] "Generic (PLEG): container finished" podID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerID="0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e" exitCode=0
Sep 17 19:08:55.863500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863128    2583 generic.go:356] "Generic (PLEG): container finished" podID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerID="a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd" exitCode=0
Sep 17 19:08:55.863500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"}
Sep 17 19:08:55.863500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"}
Sep 17 19:08:55.863500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863298    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"}
Sep 17 19:08:55.863500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:55.863341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"}
Sep 17 19:08:56.000178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.000154    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:56.046771 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-web-config\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.046928 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046793    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-out\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.046928 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-tls-assets\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.046928 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.046928 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046867    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-metrics-client-ca\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.046928 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-web\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.047194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.047194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046974    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-trusted-ca-bundle\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.047194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.046997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-cluster-tls-config\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.047194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.047032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-volume\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.047194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.047057    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-main-db\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.047194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.047083    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25jvw\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-kube-api-access-25jvw\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.047194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.047109    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\" (UID: \"d583f55a-d7ea-4ade-9b4d-3be07a8da97a\") "
Sep 17 19:08:56.048585 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.047761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:56.048585 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.048359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:56.048944 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.048886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:56.050776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.050737    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:56.051653 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.051629    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-out" (OuterVolumeSpecName: "config-out") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:56.051942 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.051751    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:56.052070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.051881    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:56.052070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.051952    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:56.052599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.052569    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-kube-api-access-25jvw" (OuterVolumeSpecName: "kube-api-access-25jvw") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "kube-api-access-25jvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:56.052790 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.052761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-volume" (OuterVolumeSpecName: "config-volume") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:56.053142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.053118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:56.056899 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.056870    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:56.062303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.062270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-web-config" (OuterVolumeSpecName: "web-config") pod "d583f55a-d7ea-4ade-9b4d-3be07a8da97a" (UID: "d583f55a-d7ea-4ade-9b4d-3be07a8da97a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:56.148071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.147996    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148023    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148034    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-cluster-tls-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148042    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-volume\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148052    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-alertmanager-main-db\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148060    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25jvw\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-kube-api-access-25jvw\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148068    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148078    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-web-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148086    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-config-out\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148093    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-tls-assets\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148101    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-main-tls\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148108    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-metrics-client-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.148367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.148118    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d583f55a-d7ea-4ade-9b4d-3be07a8da97a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:08:56.870582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.870552    2583 generic.go:356] "Generic (PLEG): container finished" podID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerID="d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617" exitCode=0
Sep 17 19:08:56.870582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.870577    2583 generic.go:356] "Generic (PLEG): container finished" podID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerID="9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac" exitCode=0
Sep 17 19:08:56.871080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.870627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"}
Sep 17 19:08:56.871080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.870641    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:56.871080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.870757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"}
Sep 17 19:08:56.871080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.870803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d583f55a-d7ea-4ade-9b4d-3be07a8da97a","Type":"ContainerDied","Data":"0d2674f430df7484a23fd2273e4d3f48f83edfc21444021cdbac434500fa9482"}
Sep 17 19:08:56.871080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.870827    2583 scope.go:117] "RemoveContainer" containerID="5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"
Sep 17 19:08:56.879077 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.878913    2583 scope.go:117] "RemoveContainer" containerID="d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"
Sep 17 19:08:56.885754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.885736    2583 scope.go:117] "RemoveContainer" containerID="44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"
Sep 17 19:08:56.892106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.892090    2583 scope.go:117] "RemoveContainer" containerID="9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"
Sep 17 19:08:56.895184 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.895163    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:08:56.899053 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.899029    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:08:56.900049 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.900031    2583 scope.go:117] "RemoveContainer" containerID="0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"
Sep 17 19:08:56.906832 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.906815    2583 scope.go:117] "RemoveContainer" containerID="a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"
Sep 17 19:08:56.913404 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.913382    2583 scope.go:117] "RemoveContainer" containerID="344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d"
Sep 17 19:08:56.919970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.919953    2583 scope.go:117] "RemoveContainer" containerID="5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"
Sep 17 19:08:56.920276 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:56.920257    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a\": container with ID starting with 5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a not found: ID does not exist" containerID="5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"
Sep 17 19:08:56.920322 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.920286    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"} err="failed to get container status \"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a\": rpc error: code = NotFound desc = could not find container \"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a\": container with ID starting with 5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a not found: ID does not exist"
Sep 17 19:08:56.920322 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.920305    2583 scope.go:117] "RemoveContainer" containerID="d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"
Sep 17 19:08:56.920556 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:56.920541    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617\": container with ID starting with d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617 not found: ID does not exist" containerID="d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"
Sep 17 19:08:56.920600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.920559    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"} err="failed to get container status \"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617\": rpc error: code = NotFound desc = could not find container \"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617\": container with ID starting with d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617 not found: ID does not exist"
Sep 17 19:08:56.920600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.920572    2583 scope.go:117] "RemoveContainer" containerID="44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"
Sep 17 19:08:56.920805 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:56.920788    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2\": container with ID starting with 44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2 not found: ID does not exist" containerID="44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"
Sep 17 19:08:56.920848 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.920811    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"} err="failed to get container status \"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2\": rpc error: code = NotFound desc = could not find container \"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2\": container with ID starting with 44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2 not found: ID does not exist"
Sep 17 19:08:56.920848 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.920826    2583 scope.go:117] "RemoveContainer" containerID="9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"
Sep 17 19:08:56.921042 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:56.921026    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac\": container with ID starting with 9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac not found: ID does not exist" containerID="9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"
Sep 17 19:08:56.921081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921046    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"} err="failed to get container status \"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac\": rpc error: code = NotFound desc = could not find container \"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac\": container with ID starting with 9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac not found: ID does not exist"
Sep 17 19:08:56.921081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921058    2583 scope.go:117] "RemoveContainer" containerID="0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"
Sep 17 19:08:56.921307 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:56.921273    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e\": container with ID starting with 0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e not found: ID does not exist" containerID="0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"
Sep 17 19:08:56.921347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921313    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"} err="failed to get container status \"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e\": rpc error: code = NotFound desc = could not find container \"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e\": container with ID starting with 0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e not found: ID does not exist"
Sep 17 19:08:56.921347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921329    2583 scope.go:117] "RemoveContainer" containerID="a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"
Sep 17 19:08:56.921567 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:56.921553    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd\": container with ID starting with a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd not found: ID does not exist" containerID="a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"
Sep 17 19:08:56.921610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921573    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"} err="failed to get container status \"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd\": rpc error: code = NotFound desc = could not find container \"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd\": container with ID starting with a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd not found: ID does not exist"
Sep 17 19:08:56.921610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921586    2583 scope.go:117] "RemoveContainer" containerID="344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d"
Sep 17 19:08:56.921825 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:08:56.921810    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d\": container with ID starting with 344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d not found: ID does not exist" containerID="344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d"
Sep 17 19:08:56.921871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921829    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d"} err="failed to get container status \"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d\": rpc error: code = NotFound desc = could not find container \"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d\": container with ID starting with 344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d not found: ID does not exist"
Sep 17 19:08:56.921871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.921841    2583 scope.go:117] "RemoveContainer" containerID="5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"
Sep 17 19:08:56.922049 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922032    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a"} err="failed to get container status \"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a\": rpc error: code = NotFound desc = could not find container \"5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a\": container with ID starting with 5e342fe9ecad30e065ac81e0030d04af701afabf169d77c78ba4a312289a302a not found: ID does not exist"
Sep 17 19:08:56.922090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922050    2583 scope.go:117] "RemoveContainer" containerID="d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"
Sep 17 19:08:56.922279 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922260    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617"} err="failed to get container status \"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617\": rpc error: code = NotFound desc = could not find container \"d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617\": container with ID starting with d091c62be09b184950ef4aa2290aecb77b3ca7e74758f67c25eb26d6cea78617 not found: ID does not exist"
Sep 17 19:08:56.922323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922280    2583 scope.go:117] "RemoveContainer" containerID="44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"
Sep 17 19:08:56.922513 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922496    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2"} err="failed to get container status \"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2\": rpc error: code = NotFound desc = could not find container \"44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2\": container with ID starting with 44b79101969b00701372a86fd1aa9c9c573fba9cefa9c75a7031ea1a7262eae2 not found: ID does not exist"
Sep 17 19:08:56.922559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922515    2583 scope.go:117] "RemoveContainer" containerID="9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"
Sep 17 19:08:56.922745 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922728    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac"} err="failed to get container status \"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac\": rpc error: code = NotFound desc = could not find container \"9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac\": container with ID starting with 9b84dd8a85bf8a747645b47416e4c15be879b47bc51a025eec4d6baa5b8627ac not found: ID does not exist"
Sep 17 19:08:56.922791 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922746    2583 scope.go:117] "RemoveContainer" containerID="0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"
Sep 17 19:08:56.922944 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922928    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e"} err="failed to get container status \"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e\": rpc error: code = NotFound desc = could not find container \"0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e\": container with ID starting with 0ba6e98a7897fd787655587619d8b0da628ad7629cd5522e60f8f86d0645b82e not found: ID does not exist"
Sep 17 19:08:56.922989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.922945    2583 scope.go:117] "RemoveContainer" containerID="a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"
Sep 17 19:08:56.923169 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.923148    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd"} err="failed to get container status \"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd\": rpc error: code = NotFound desc = could not find container \"a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd\": container with ID starting with a039c6b0784428403b1e1d38f00857f8bcc8d9f3d6e2caa7b0c0b2f63ec79ebd not found: ID does not exist"
Sep 17 19:08:56.923207 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.923171    2583 scope.go:117] "RemoveContainer" containerID="344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d"
Sep 17 19:08:56.923388 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.923372    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d"} err="failed to get container status \"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d\": rpc error: code = NotFound desc = could not find container \"344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d\": container with ID starting with 344568fec9e4c8c4db608c9a5da1146194879840bad03c5c8b0335cf89385a7d not found: ID does not exist"
Sep 17 19:08:56.940867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.940837    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:08:56.941106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941093    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="alertmanager"
Sep 17 19:08:56.941150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941109    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="alertmanager"
Sep 17 19:08:56.941150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941117    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="init-config-reloader"
Sep 17 19:08:56.941150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941122    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="init-config-reloader"
Sep 17 19:08:56.941150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941131    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy"
Sep 17 19:08:56.941150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941137    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy"
Sep 17 19:08:56.941150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941142    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-metric"
Sep 17 19:08:56.941150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941147    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-metric"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941156    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="config-reloader"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="config-reloader"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941178    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-web"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941183    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-web"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941188    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="prom-label-proxy"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941193    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="prom-label-proxy"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941201    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a691241d-1495-480a-a02d-3df4bc18299d" containerName="console"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941207    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a691241d-1495-480a-a02d-3df4bc18299d" containerName="console"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941247    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a691241d-1495-480a-a02d-3df4bc18299d" containerName="console"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941261    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="prom-label-proxy"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941267    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-web"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941272    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941277    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="kube-rbac-proxy-metric"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941284    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="alertmanager"
Sep 17 19:08:56.941344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.941289    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" containerName="config-reloader"
Sep 17 19:08:56.944955 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.944940    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:56.947734 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.947714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 19:08:56.947734 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.947720    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 19:08:56.947887 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.947720    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 19:08:56.947887 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.947720    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 19:08:56.948309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.948281    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 19:08:56.948508 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.948490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 19:08:56.948586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.948498    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 19:08:56.948586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.948536    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 19:08:56.948586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.948578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jwl5c\""
Sep 17 19:08:56.953599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.953578    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 19:08:56.961413 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:56.961393    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:08:57.054235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-web-config\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/090fc964-f14c-4de7-9ac4-5b87652938e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/090fc964-f14c-4de7-9ac4-5b87652938e3-config-out\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/090fc964-f14c-4de7-9ac4-5b87652938e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbg4w\" (UniqueName: \"kubernetes.io/projected/090fc964-f14c-4de7-9ac4-5b87652938e3-kube-api-access-qbg4w\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090fc964-f14c-4de7-9ac4-5b87652938e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/090fc964-f14c-4de7-9ac4-5b87652938e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.054674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.054593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.155986 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.155912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.155986 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.155956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.155986 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.155971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-web-config\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156167 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/090fc964-f14c-4de7-9ac4-5b87652938e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156167 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/090fc964-f14c-4de7-9ac4-5b87652938e3-config-out\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156167 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/090fc964-f14c-4de7-9ac4-5b87652938e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg4w\" (UniqueName: \"kubernetes.io/projected/090fc964-f14c-4de7-9ac4-5b87652938e3-kube-api-access-qbg4w\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090fc964-f14c-4de7-9ac4-5b87652938e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156611 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/090fc964-f14c-4de7-9ac4-5b87652938e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.156681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.156659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/090fc964-f14c-4de7-9ac4-5b87652938e3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.157435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.157129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/090fc964-f14c-4de7-9ac4-5b87652938e3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.158204 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.158178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090fc964-f14c-4de7-9ac4-5b87652938e3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.159418 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.159390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.159511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.159481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/090fc964-f14c-4de7-9ac4-5b87652938e3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.159511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.159487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-web-config\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.159629 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.159588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-config-volume\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.159677 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.159646    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.159837 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.159819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/090fc964-f14c-4de7-9ac4-5b87652938e3-config-out\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.159903 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.159874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.160905 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.160885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.161819 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.161799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/090fc964-f14c-4de7-9ac4-5b87652938e3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.165257 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.165240    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbg4w\" (UniqueName: \"kubernetes.io/projected/090fc964-f14c-4de7-9ac4-5b87652938e3-kube-api-access-qbg4w\") pod \"alertmanager-main-0\" (UID: \"090fc964-f14c-4de7-9ac4-5b87652938e3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.254576 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.254540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:08:57.532268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.532186    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:08:57.535783 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:08:57.535756    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090fc964_f14c_4de7_9ac4_5b87652938e3.slice/crio-3d4fbbb22489e18899d3b72b56851b6e27c15586d3bc244b39c2237927cf52f3 WatchSource:0}: Error finding container 3d4fbbb22489e18899d3b72b56851b6e27c15586d3bc244b39c2237927cf52f3: Status 404 returned error can't find the container with id 3d4fbbb22489e18899d3b72b56851b6e27c15586d3bc244b39c2237927cf52f3
Sep 17 19:08:57.876963 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.876927    2583 generic.go:356] "Generic (PLEG): container finished" podID="090fc964-f14c-4de7-9ac4-5b87652938e3" containerID="d949e78a22490cb8ada14773a371e55c8747c980f01ef5b6a465eabdb7141797" exitCode=0
Sep 17 19:08:57.877314 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.876983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerDied","Data":"d949e78a22490cb8ada14773a371e55c8747c980f01ef5b6a465eabdb7141797"}
Sep 17 19:08:57.877314 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:57.877039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerStarted","Data":"3d4fbbb22489e18899d3b72b56851b6e27c15586d3bc244b39c2237927cf52f3"}
Sep 17 19:08:58.253410 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.253381    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d583f55a-d7ea-4ade-9b4d-3be07a8da97a" path="/var/lib/kubelet/pods/d583f55a-d7ea-4ade-9b4d-3be07a8da97a/volumes"
Sep 17 19:08:58.754119 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.754082    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"]
Sep 17 19:08:58.756791 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.756774    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.759139 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.759122    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 19:08:58.759453 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.759429    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 19:08:58.759559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.759429    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 19:08:58.759559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.759493    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 19:08:58.759668 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.759604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9xpcx\""
Sep 17 19:08:58.759740 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.759720    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 19:08:58.765062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.765040    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 19:08:58.774017 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.773992    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"]
Sep 17 19:08:58.876914 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.876884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-federate-client-tls\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.877080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.876934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-telemeter-client-tls\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.877080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.876963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-serving-certs-ca-bundle\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.877080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.876987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-metrics-client-ca\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.877080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.877028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.877080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.877057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.877485 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.877118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-secret-telemeter-client\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.877485 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.877157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6mp\" (UniqueName: \"kubernetes.io/projected/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-kube-api-access-pz6mp\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.883485 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.883460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerStarted","Data":"9cd5e1c6d5172e0eee38a2237878006c8150d049c0f0f45d90ea4337a0ddabee"}
Sep 17 19:08:58.883616 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.883533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerStarted","Data":"fa123398947ad975401537829f3842d70387f0edef3e5ddb844143726d9c72d9"}
Sep 17 19:08:58.883616 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.883578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerStarted","Data":"00a46485fcc404bd2b96a3eb7f118599963cfbc0fd9d7a300ee04121f4f36669"}
Sep 17 19:08:58.883747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.883624    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerStarted","Data":"768efe44b5d7cee06b3768f456c3b55ee6d1d89d9d5c2b6c005981d65048480e"}
Sep 17 19:08:58.883747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.883667    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerStarted","Data":"52e57e3b2457dd86745b86261b6e6f1a97d73432e51d8e400537b0acc32f169a"}
Sep 17 19:08:58.883843 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.883743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"090fc964-f14c-4de7-9ac4-5b87652938e3","Type":"ContainerStarted","Data":"e014cc50f29a4989db97f44fff688d0393172f11edb17429bef44bfb27763d74"}
Sep 17 19:08:58.909484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.909438    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.894628174 podStartE2EDuration="2.909423462s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.795288ms" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:58.907465134 +0000 UTC m=+140.955344992" watchObservedRunningTime="2026-09-17 19:08:58.909423462 +0000 UTC m=+140.957303346"
Sep 17 19:08:58.978220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.978220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-secret-telemeter-client\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.978434 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6mp\" (UniqueName: \"kubernetes.io/projected/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-kube-api-access-pz6mp\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.978434 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-federate-client-tls\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.978548 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-telemeter-client-tls\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.978600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-serving-certs-ca-bundle\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.978656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978610    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-metrics-client-ca\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.978656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.978643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.979500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.979460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-metrics-client-ca\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.979500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.979460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-serving-certs-ca-bundle\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.980261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.980237    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.981382 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.981357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-telemeter-client-tls\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.981382 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.981375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-federate-client-tls\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.981508 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.981477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.981846 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.981825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-secret-telemeter-client\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:58.987369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:58.987348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6mp\" (UniqueName: \"kubernetes.io/projected/19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7-kube-api-access-pz6mp\") pod \"telemeter-client-7f7957bd96-ptrpg\" (UID: \"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7\") " pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:59.067925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:59.067898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"
Sep 17 19:08:59.346145 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:59.346046    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f7957bd96-ptrpg"]
Sep 17 19:08:59.348547 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:08:59.348506    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b0be20_4e4f_44d1_82a7_c95f0cc1fbb7.slice/crio-bf34e33a43d67d579a718e279e319afaee927c222e188111c659eb0073dd1e63 WatchSource:0}: Error finding container bf34e33a43d67d579a718e279e319afaee927c222e188111c659eb0073dd1e63: Status 404 returned error can't find the container with id bf34e33a43d67d579a718e279e319afaee927c222e188111c659eb0073dd1e63
Sep 17 19:08:59.889588 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:08:59.889545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg" event={"ID":"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7","Type":"ContainerStarted","Data":"bf34e33a43d67d579a718e279e319afaee927c222e188111c659eb0073dd1e63"}
Sep 17 19:09:01.899466 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:01.899424    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg" event={"ID":"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7","Type":"ContainerStarted","Data":"66a207756cb78b957fdd54482fdd22297066e356abea291279ff94e2c3de049f"}
Sep 17 19:09:01.899919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:01.899483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg" event={"ID":"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7","Type":"ContainerStarted","Data":"98634167769dafcb8d401ca0ffeeefad4c39b3d345d54e6f52967e3aa8756838"}
Sep 17 19:09:01.899919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:01.899501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg" event={"ID":"19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7","Type":"ContainerStarted","Data":"7b54b0b049ded291b70dbd05ed7b46d3c95f8ec3c50e9bc7ebf8b73019f794d6"}
Sep 17 19:09:01.926637 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:01.926588    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f7957bd96-ptrpg" podStartSLOduration=2.199126841 podStartE2EDuration="3.926572821s" totalImagesPullingTime="1.72744598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:01.925976481 +0000 UTC m=+143.973856338" watchObservedRunningTime="2026-09-17 19:09:01.926572821 +0000 UTC m=+143.974452678"
Sep 17 19:09:02.023408 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.023378    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7948f845fb-rvvqf"]
Sep 17 19:09:02.025669 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.025653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.039840 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.039813    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7948f845fb-rvvqf"]
Sep 17 19:09:02.107648 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.107622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-oauth-config\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.107818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.107662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-trusted-ca-bundle\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.107818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.107707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-console-config\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.107818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.107765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-oauth-serving-cert\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.107818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.107803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-service-ca\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.107952 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.107858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-serving-cert\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.107952 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.107908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xd2j\" (UniqueName: \"kubernetes.io/projected/dd346e81-495e-453e-bf86-1dadff751195-kube-api-access-7xd2j\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.208492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.208433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xd2j\" (UniqueName: \"kubernetes.io/projected/dd346e81-495e-453e-bf86-1dadff751195-kube-api-access-7xd2j\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.208492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.208464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-oauth-config\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.208492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.208491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-trusted-ca-bundle\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.208722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.208515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-console-config\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.208722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.208643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-oauth-serving-cert\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.208722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.208705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-service-ca\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.208862 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.208798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-serving-cert\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.209293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.209261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-console-config\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.209413 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.209344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-oauth-serving-cert\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.209512 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.209491    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-service-ca\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.209599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.209579    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-trusted-ca-bundle\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.211162 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.211137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-oauth-config\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.211333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.211317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-serving-cert\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.223871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.223854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xd2j\" (UniqueName: \"kubernetes.io/projected/dd346e81-495e-453e-bf86-1dadff751195-kube-api-access-7xd2j\") pod \"console-7948f845fb-rvvqf\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") " pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.334795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.334756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:02.608715 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.608665    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7948f845fb-rvvqf"]
Sep 17 19:09:02.611706 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:09:02.611663    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd346e81_495e_453e_bf86_1dadff751195.slice/crio-fc8de6a93091a9ab5ef8eb5f6241ac4ee94c5cf01703de5ed9e3f490d099d6ac WatchSource:0}: Error finding container fc8de6a93091a9ab5ef8eb5f6241ac4ee94c5cf01703de5ed9e3f490d099d6ac: Status 404 returned error can't find the container with id fc8de6a93091a9ab5ef8eb5f6241ac4ee94c5cf01703de5ed9e3f490d099d6ac
Sep 17 19:09:02.905122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.905024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7948f845fb-rvvqf" event={"ID":"dd346e81-495e-453e-bf86-1dadff751195","Type":"ContainerStarted","Data":"3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302"}
Sep 17 19:09:02.905494 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.905119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7948f845fb-rvvqf" event={"ID":"dd346e81-495e-453e-bf86-1dadff751195","Type":"ContainerStarted","Data":"fc8de6a93091a9ab5ef8eb5f6241ac4ee94c5cf01703de5ed9e3f490d099d6ac"}
Sep 17 19:09:02.922127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:02.922079    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7948f845fb-rvvqf" podStartSLOduration=0.922064026 podStartE2EDuration="922.064026ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:02.921622807 +0000 UTC m=+144.969502688" watchObservedRunningTime="2026-09-17 19:09:02.922064026 +0000 UTC m=+144.969943883"
Sep 17 19:09:12.335112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:12.335072    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:12.335572 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:12.335135    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:12.339848 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:12.339828    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:12.949215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:12.949188    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:09:12.999250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:12.999220    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f97889b4-ljpj2"]
Sep 17 19:09:38.018675 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.018608    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79f97889b4-ljpj2" podUID="04bf16b4-87b1-4dd3-8f11-104d036a16cf" containerName="console" containerID="cri-o://4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f" gracePeriod=15
Sep 17 19:09:38.246868 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.246846    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:09:38.297348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-oauth-serving-cert\") pod \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") "
Sep 17 19:09:38.297475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297368    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-config\") pod \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") "
Sep 17 19:09:38.297475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-serving-cert\") pod \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") "
Sep 17 19:09:38.297475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297433    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-trusted-ca-bundle\") pod \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") "
Sep 17 19:09:38.297639 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-oauth-config\") pod \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") "
Sep 17 19:09:38.297639 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297575    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-service-ca\") pod \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") "
Sep 17 19:09:38.297639 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297600    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4snk4\" (UniqueName: \"kubernetes.io/projected/04bf16b4-87b1-4dd3-8f11-104d036a16cf-kube-api-access-4snk4\") pod \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\" (UID: \"04bf16b4-87b1-4dd3-8f11-104d036a16cf\") "
Sep 17 19:09:38.297904 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-config" (OuterVolumeSpecName: "console-config") pod "04bf16b4-87b1-4dd3-8f11-104d036a16cf" (UID: "04bf16b4-87b1-4dd3-8f11-104d036a16cf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:38.297904 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04bf16b4-87b1-4dd3-8f11-104d036a16cf" (UID: "04bf16b4-87b1-4dd3-8f11-104d036a16cf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:38.298048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297888    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "04bf16b4-87b1-4dd3-8f11-104d036a16cf" (UID: "04bf16b4-87b1-4dd3-8f11-104d036a16cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:38.298048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.297954    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-service-ca" (OuterVolumeSpecName: "service-ca") pod "04bf16b4-87b1-4dd3-8f11-104d036a16cf" (UID: "04bf16b4-87b1-4dd3-8f11-104d036a16cf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:38.299812 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.299786    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bf16b4-87b1-4dd3-8f11-104d036a16cf-kube-api-access-4snk4" (OuterVolumeSpecName: "kube-api-access-4snk4") pod "04bf16b4-87b1-4dd3-8f11-104d036a16cf" (UID: "04bf16b4-87b1-4dd3-8f11-104d036a16cf"). InnerVolumeSpecName "kube-api-access-4snk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:38.299892 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.299813    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04bf16b4-87b1-4dd3-8f11-104d036a16cf" (UID: "04bf16b4-87b1-4dd3-8f11-104d036a16cf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:38.299892 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.299828    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04bf16b4-87b1-4dd3-8f11-104d036a16cf" (UID: "04bf16b4-87b1-4dd3-8f11-104d036a16cf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:38.398313 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.398286    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:09:38.398313 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.398308    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:09:38.398313 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.398317    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:09:38.398481 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.398325    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04bf16b4-87b1-4dd3-8f11-104d036a16cf-console-oauth-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:09:38.398481 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.398334    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:09:38.398481 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.398342    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snk4\" (UniqueName: \"kubernetes.io/projected/04bf16b4-87b1-4dd3-8f11-104d036a16cf-kube-api-access-4snk4\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:09:38.398481 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:38.398350    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04bf16b4-87b1-4dd3-8f11-104d036a16cf-oauth-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:09:39.044454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.044420    2583 generic.go:356] "Generic (PLEG): container finished" podID="04bf16b4-87b1-4dd3-8f11-104d036a16cf" containerID="4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f" exitCode=0
Sep 17 19:09:39.044850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.044459    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f97889b4-ljpj2"
Sep 17 19:09:39.044850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.044488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f97889b4-ljpj2" event={"ID":"04bf16b4-87b1-4dd3-8f11-104d036a16cf","Type":"ContainerDied","Data":"4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f"}
Sep 17 19:09:39.044850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.044528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f97889b4-ljpj2" event={"ID":"04bf16b4-87b1-4dd3-8f11-104d036a16cf","Type":"ContainerDied","Data":"bfd9c3d339cb5212c681217bfe0355c15707b883939a605f8ef94a82c37d07f6"}
Sep 17 19:09:39.044850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.044548    2583 scope.go:117] "RemoveContainer" containerID="4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f"
Sep 17 19:09:39.053211 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.053194    2583 scope.go:117] "RemoveContainer" containerID="4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f"
Sep 17 19:09:39.053460 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:09:39.053440    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f\": container with ID starting with 4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f not found: ID does not exist" containerID="4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f"
Sep 17 19:09:39.053509 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.053469    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f"} err="failed to get container status \"4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f\": rpc error: code = NotFound desc = could not find container \"4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f\": container with ID starting with 4cd8ac2f847200f5d6259f843082ad92a63f6c730b01a525d880751540bbd32f not found: ID does not exist"
Sep 17 19:09:39.068275 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.068250    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f97889b4-ljpj2"]
Sep 17 19:09:39.070525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:39.070501    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f97889b4-ljpj2"]
Sep 17 19:09:40.253348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:09:40.253309    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bf16b4-87b1-4dd3-8f11-104d036a16cf" path="/var/lib/kubelet/pods/04bf16b4-87b1-4dd3-8f11-104d036a16cf/volumes"
Sep 17 19:10:31.444230 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.444192    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f4585dc68-dxqpz"]
Sep 17 19:10:31.444717 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.444538    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bf16b4-87b1-4dd3-8f11-104d036a16cf" containerName="console"
Sep 17 19:10:31.444717 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.444572    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bf16b4-87b1-4dd3-8f11-104d036a16cf" containerName="console"
Sep 17 19:10:31.444717 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.444626    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bf16b4-87b1-4dd3-8f11-104d036a16cf" containerName="console"
Sep 17 19:10:31.446403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.446385    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.464791 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.464758    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4585dc68-dxqpz"]
Sep 17 19:10:31.574007 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.573975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hbd\" (UniqueName: \"kubernetes.io/projected/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-kube-api-access-k5hbd\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.574007 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.574011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-config\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.574220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.574036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-serving-cert\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.574220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.574061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-oauth-config\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.574220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.574076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-oauth-serving-cert\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.574220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.574161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-trusted-ca-bundle\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.574220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.574200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-service-ca\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.674730 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.674679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-service-ca\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.674857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.674744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hbd\" (UniqueName: \"kubernetes.io/projected/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-kube-api-access-k5hbd\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.674857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.674763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-config\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.674857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.674785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-serving-cert\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.674857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.674808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-oauth-config\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.674857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.674822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-oauth-serving-cert\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.675129 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.674882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-trusted-ca-bundle\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.675576 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.675541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-oauth-serving-cert\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.675576 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.675556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-service-ca\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.675732 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.675556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-config\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.675975 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.675954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-trusted-ca-bundle\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.677323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.677302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-oauth-config\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.677428 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.677411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-serving-cert\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.684329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.684309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hbd\" (UniqueName: \"kubernetes.io/projected/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-kube-api-access-k5hbd\") pod \"console-6f4585dc68-dxqpz\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") " pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:31.755410 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:31.755339    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:32.029933 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:32.029908    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4585dc68-dxqpz"]
Sep 17 19:10:32.032410 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:10:32.032383    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde25ad3_adfe_4db0_8bae_af4c2f6d49a2.slice/crio-99da12e90554864c2545a0eed959b59893d654c47b66bc289e41f4f4b5d7bd73 WatchSource:0}: Error finding container 99da12e90554864c2545a0eed959b59893d654c47b66bc289e41f4f4b5d7bd73: Status 404 returned error can't find the container with id 99da12e90554864c2545a0eed959b59893d654c47b66bc289e41f4f4b5d7bd73
Sep 17 19:10:32.234251 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:32.234206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4585dc68-dxqpz" event={"ID":"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2","Type":"ContainerStarted","Data":"48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28"}
Sep 17 19:10:32.234403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:32.234267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4585dc68-dxqpz" event={"ID":"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2","Type":"ContainerStarted","Data":"99da12e90554864c2545a0eed959b59893d654c47b66bc289e41f4f4b5d7bd73"}
Sep 17 19:10:32.261367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:32.261318    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f4585dc68-dxqpz" podStartSLOduration=1.2613001719999999 podStartE2EDuration="1.261300172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:32.260482611 +0000 UTC m=+234.308362496" watchObservedRunningTime="2026-09-17 19:10:32.261300172 +0000 UTC m=+234.309180029"
Sep 17 19:10:41.756010 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:41.755976    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:41.756488 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:41.756041    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:41.760235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:41.760217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:42.271894 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:42.271862    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:10:42.334717 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:10:42.334667    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7948f845fb-rvvqf"]
Sep 17 19:11:03.912930 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:03.912893    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"]
Sep 17 19:11:03.915466 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:03.915446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:03.918595 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:03.918571    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pgp8f\""
Sep 17 19:11:03.918860 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:03.918843    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 19:11:03.919633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:03.919619    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 19:11:03.930112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:03.930086    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"]
Sep 17 19:11:04.011284 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.011255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.011394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.011303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86qq\" (UniqueName: \"kubernetes.io/projected/ff1b418f-cc60-41b9-891c-fa62e2e260f6-kube-api-access-m86qq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.011447 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.011381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.112401 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.112371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.112519 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.112417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86qq\" (UniqueName: \"kubernetes.io/projected/ff1b418f-cc60-41b9-891c-fa62e2e260f6-kube-api-access-m86qq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.112519 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.112463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.112773 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.112749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.112856 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.112821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.123314 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.123289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86qq\" (UniqueName: \"kubernetes.io/projected/ff1b418f-cc60-41b9-891c-fa62e2e260f6-kube-api-access-m86qq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.225133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.225058    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:04.498267 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:04.498243    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"]
Sep 17 19:11:04.500750 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:11:04.500721    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1b418f_cc60_41b9_891c_fa62e2e260f6.slice/crio-c5c5db5bb83744af5c99c8de00bde3823fe3e2cf9a0aabca1991a557a0fbc687 WatchSource:0}: Error finding container c5c5db5bb83744af5c99c8de00bde3823fe3e2cf9a0aabca1991a557a0fbc687: Status 404 returned error can't find the container with id c5c5db5bb83744af5c99c8de00bde3823fe3e2cf9a0aabca1991a557a0fbc687
Sep 17 19:11:05.349253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:05.349217    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" event={"ID":"ff1b418f-cc60-41b9-891c-fa62e2e260f6","Type":"ContainerStarted","Data":"c5c5db5bb83744af5c99c8de00bde3823fe3e2cf9a0aabca1991a557a0fbc687"}
Sep 17 19:11:07.354285 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.354234    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7948f845fb-rvvqf" podUID="dd346e81-495e-453e-bf86-1dadff751195" containerName="console" containerID="cri-o://3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302" gracePeriod=15
Sep 17 19:11:07.591705 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.591667    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:11:07.638822 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.638762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-oauth-serving-cert\") pod \"dd346e81-495e-453e-bf86-1dadff751195\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") "
Sep 17 19:11:07.638822 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.638810    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xd2j\" (UniqueName: \"kubernetes.io/projected/dd346e81-495e-453e-bf86-1dadff751195-kube-api-access-7xd2j\") pod \"dd346e81-495e-453e-bf86-1dadff751195\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") "
Sep 17 19:11:07.638999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.638839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-trusted-ca-bundle\") pod \"dd346e81-495e-453e-bf86-1dadff751195\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") "
Sep 17 19:11:07.638999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.638865    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-console-config\") pod \"dd346e81-495e-453e-bf86-1dadff751195\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") "
Sep 17 19:11:07.638999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.638890    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-service-ca\") pod \"dd346e81-495e-453e-bf86-1dadff751195\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") "
Sep 17 19:11:07.638999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.638925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-serving-cert\") pod \"dd346e81-495e-453e-bf86-1dadff751195\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") "
Sep 17 19:11:07.638999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.638997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-oauth-config\") pod \"dd346e81-495e-453e-bf86-1dadff751195\" (UID: \"dd346e81-495e-453e-bf86-1dadff751195\") "
Sep 17 19:11:07.639235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.639175    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dd346e81-495e-453e-bf86-1dadff751195" (UID: "dd346e81-495e-453e-bf86-1dadff751195"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:07.639293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.639235    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-oauth-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:07.639293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.639270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dd346e81-495e-453e-bf86-1dadff751195" (UID: "dd346e81-495e-453e-bf86-1dadff751195"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:07.639416 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.639312    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-service-ca" (OuterVolumeSpecName: "service-ca") pod "dd346e81-495e-453e-bf86-1dadff751195" (UID: "dd346e81-495e-453e-bf86-1dadff751195"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:07.639752 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.639720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-console-config" (OuterVolumeSpecName: "console-config") pod "dd346e81-495e-453e-bf86-1dadff751195" (UID: "dd346e81-495e-453e-bf86-1dadff751195"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:07.641304 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.641270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd346e81-495e-453e-bf86-1dadff751195-kube-api-access-7xd2j" (OuterVolumeSpecName: "kube-api-access-7xd2j") pod "dd346e81-495e-453e-bf86-1dadff751195" (UID: "dd346e81-495e-453e-bf86-1dadff751195"). InnerVolumeSpecName "kube-api-access-7xd2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:07.641445 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.641410    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dd346e81-495e-453e-bf86-1dadff751195" (UID: "dd346e81-495e-453e-bf86-1dadff751195"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:07.642052 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.642025    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dd346e81-495e-453e-bf86-1dadff751195" (UID: "dd346e81-495e-453e-bf86-1dadff751195"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:07.739750 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.739721    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xd2j\" (UniqueName: \"kubernetes.io/projected/dd346e81-495e-453e-bf86-1dadff751195-kube-api-access-7xd2j\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:07.739750 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.739747    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:07.739921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.739761    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-console-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:07.739921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.739772    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dd346e81-495e-453e-bf86-1dadff751195-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:07.739921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.739783    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:07.739921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:07.739794    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dd346e81-495e-453e-bf86-1dadff751195-console-oauth-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:08.362207 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.362176    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd346e81-495e-453e-bf86-1dadff751195" containerID="3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302" exitCode=0
Sep 17 19:11:08.362591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.362236    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7948f845fb-rvvqf"
Sep 17 19:11:08.362591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.362223    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7948f845fb-rvvqf" event={"ID":"dd346e81-495e-453e-bf86-1dadff751195","Type":"ContainerDied","Data":"3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302"}
Sep 17 19:11:08.362591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.362534    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7948f845fb-rvvqf" event={"ID":"dd346e81-495e-453e-bf86-1dadff751195","Type":"ContainerDied","Data":"fc8de6a93091a9ab5ef8eb5f6241ac4ee94c5cf01703de5ed9e3f490d099d6ac"}
Sep 17 19:11:08.362591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.362570    2583 scope.go:117] "RemoveContainer" containerID="3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302"
Sep 17 19:11:08.372970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.372907    2583 scope.go:117] "RemoveContainer" containerID="3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302"
Sep 17 19:11:08.373259 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:08.373234    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302\": container with ID starting with 3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302 not found: ID does not exist" containerID="3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302"
Sep 17 19:11:08.373368 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.373269    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302"} err="failed to get container status \"3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302\": rpc error: code = NotFound desc = could not find container \"3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302\": container with ID starting with 3f45642a9ac57e66cfed309f35345be096c3420ee5e0ecb0fb7e97005ea8e302 not found: ID does not exist"
Sep 17 19:11:08.383284 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.383262    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7948f845fb-rvvqf"]
Sep 17 19:11:08.387137 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:08.387116    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7948f845fb-rvvqf"]
Sep 17 19:11:10.255357 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:10.255322    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd346e81-495e-453e-bf86-1dadff751195" path="/var/lib/kubelet/pods/dd346e81-495e-453e-bf86-1dadff751195/volumes"
Sep 17 19:11:11.376099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:11.376069    2583 generic.go:356] "Generic (PLEG): container finished" podID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerID="f8da22d90c23373d1c677886c8d86b6c8877bf447ee2f89e9e8644f31211c1ce" exitCode=0
Sep 17 19:11:11.376453 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:11.376141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" event={"ID":"ff1b418f-cc60-41b9-891c-fa62e2e260f6","Type":"ContainerDied","Data":"f8da22d90c23373d1c677886c8d86b6c8877bf447ee2f89e9e8644f31211c1ce"}
Sep 17 19:11:13.385389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:13.385356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" event={"ID":"ff1b418f-cc60-41b9-891c-fa62e2e260f6","Type":"ContainerStarted","Data":"3161f31ef568f1c94f9c3ac68fb105df8455d76762b76c50ff02a57711693927"}
Sep 17 19:11:14.391396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:14.391352    2583 generic.go:356] "Generic (PLEG): container finished" podID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerID="3161f31ef568f1c94f9c3ac68fb105df8455d76762b76c50ff02a57711693927" exitCode=0
Sep 17 19:11:14.391852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:14.391445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" event={"ID":"ff1b418f-cc60-41b9-891c-fa62e2e260f6","Type":"ContainerDied","Data":"3161f31ef568f1c94f9c3ac68fb105df8455d76762b76c50ff02a57711693927"}
Sep 17 19:11:21.425988 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:21.425952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" event={"ID":"ff1b418f-cc60-41b9-891c-fa62e2e260f6","Type":"ContainerStarted","Data":"b65fa34cc00e512aa11a26e2bc36f69e12ed8b3952c3006078afe9b596e4b3d6"}
Sep 17 19:11:21.446431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:21.446386    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" podStartSLOduration=2.965522356 podStartE2EDuration="18.446370978s" totalImagesPullingTime="15.480848622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:21.445088677 +0000 UTC m=+283.492968534" watchObservedRunningTime="2026-09-17 19:11:21.446370978 +0000 UTC m=+283.494250836"
Sep 17 19:11:22.432024 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:22.431985    2583 generic.go:356] "Generic (PLEG): container finished" podID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerID="b65fa34cc00e512aa11a26e2bc36f69e12ed8b3952c3006078afe9b596e4b3d6" exitCode=0
Sep 17 19:11:22.432445 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:22.432032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" event={"ID":"ff1b418f-cc60-41b9-891c-fa62e2e260f6","Type":"ContainerDied","Data":"b65fa34cc00e512aa11a26e2bc36f69e12ed8b3952c3006078afe9b596e4b3d6"}
Sep 17 19:11:23.563834 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.563811    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:23.658602 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.658578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-util\") pod \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") "
Sep 17 19:11:23.658756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.658614    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m86qq\" (UniqueName: \"kubernetes.io/projected/ff1b418f-cc60-41b9-891c-fa62e2e260f6-kube-api-access-m86qq\") pod \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") "
Sep 17 19:11:23.658756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.658669    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-bundle\") pod \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\" (UID: \"ff1b418f-cc60-41b9-891c-fa62e2e260f6\") "
Sep 17 19:11:23.659523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.659491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-bundle" (OuterVolumeSpecName: "bundle") pod "ff1b418f-cc60-41b9-891c-fa62e2e260f6" (UID: "ff1b418f-cc60-41b9-891c-fa62e2e260f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:11:23.660983 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.660961    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1b418f-cc60-41b9-891c-fa62e2e260f6-kube-api-access-m86qq" (OuterVolumeSpecName: "kube-api-access-m86qq") pod "ff1b418f-cc60-41b9-891c-fa62e2e260f6" (UID: "ff1b418f-cc60-41b9-891c-fa62e2e260f6"). InnerVolumeSpecName "kube-api-access-m86qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:23.663379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.663360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-util" (OuterVolumeSpecName: "util") pod "ff1b418f-cc60-41b9-891c-fa62e2e260f6" (UID: "ff1b418f-cc60-41b9-891c-fa62e2e260f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:11:23.759469 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.759409    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-util\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:23.759469 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.759440    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86qq\" (UniqueName: \"kubernetes.io/projected/ff1b418f-cc60-41b9-891c-fa62e2e260f6-kube-api-access-m86qq\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:23.759469 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:23.759451    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff1b418f-cc60-41b9-891c-fa62e2e260f6-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:11:24.441768 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:24.441731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg" event={"ID":"ff1b418f-cc60-41b9-891c-fa62e2e260f6","Type":"ContainerDied","Data":"c5c5db5bb83744af5c99c8de00bde3823fe3e2cf9a0aabca1991a557a0fbc687"}
Sep 17 19:11:24.441768 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:24.441757    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adtgzg"
Sep 17 19:11:24.441937 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:24.441788    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c5db5bb83744af5c99c8de00bde3823fe3e2cf9a0aabca1991a557a0fbc687"
Sep 17 19:11:26.303528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303495    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"]
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303774    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerName="extract"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303785    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerName="extract"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303795    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerName="pull"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303801    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerName="pull"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303811    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd346e81-495e-453e-bf86-1dadff751195" containerName="console"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303817    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd346e81-495e-453e-bf86-1dadff751195" containerName="console"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303830    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerName="util"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303835    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerName="util"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303876    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1b418f-cc60-41b9-891c-fa62e2e260f6" containerName="extract"
Sep 17 19:11:26.303893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.303884    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd346e81-495e-453e-bf86-1dadff751195" containerName="console"
Sep 17 19:11:26.344562 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.344535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"]
Sep 17 19:11:26.344704 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.344622    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.347360 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.347339    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 19:11:26.347574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.347558    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-lm55p\""
Sep 17 19:11:26.347658 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.347559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 19:11:26.347658 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.347563    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 19:11:26.477797 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.477772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a8dc49d1-dffd-4fc8-9920-a0173e96e689-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fnpz8\" (UID: \"a8dc49d1-dffd-4fc8-9920-a0173e96e689\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.477929 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.477826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8v9\" (UniqueName: \"kubernetes.io/projected/a8dc49d1-dffd-4fc8-9920-a0173e96e689-kube-api-access-hj8v9\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fnpz8\" (UID: \"a8dc49d1-dffd-4fc8-9920-a0173e96e689\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.578198 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.578136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a8dc49d1-dffd-4fc8-9920-a0173e96e689-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fnpz8\" (UID: \"a8dc49d1-dffd-4fc8-9920-a0173e96e689\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.578298 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.578208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8v9\" (UniqueName: \"kubernetes.io/projected/a8dc49d1-dffd-4fc8-9920-a0173e96e689-kube-api-access-hj8v9\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fnpz8\" (UID: \"a8dc49d1-dffd-4fc8-9920-a0173e96e689\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.580503 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.580482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a8dc49d1-dffd-4fc8-9920-a0173e96e689-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fnpz8\" (UID: \"a8dc49d1-dffd-4fc8-9920-a0173e96e689\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.587117 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.587099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8v9\" (UniqueName: \"kubernetes.io/projected/a8dc49d1-dffd-4fc8-9920-a0173e96e689-kube-api-access-hj8v9\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fnpz8\" (UID: \"a8dc49d1-dffd-4fc8-9920-a0173e96e689\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.654548 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.654531    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:26.955238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:26.955213    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"]
Sep 17 19:11:26.957374 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:11:26.957340    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dc49d1_dffd_4fc8_9920_a0173e96e689.slice/crio-7d3196fa50466dfaa2f298d0b23c7cabb6163ddd28701e84519bd6d2f83cc7f9 WatchSource:0}: Error finding container 7d3196fa50466dfaa2f298d0b23c7cabb6163ddd28701e84519bd6d2f83cc7f9: Status 404 returned error can't find the container with id 7d3196fa50466dfaa2f298d0b23c7cabb6163ddd28701e84519bd6d2f83cc7f9
Sep 17 19:11:27.453960 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:27.453928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8" event={"ID":"a8dc49d1-dffd-4fc8-9920-a0173e96e689","Type":"ContainerStarted","Data":"7d3196fa50466dfaa2f298d0b23c7cabb6163ddd28701e84519bd6d2f83cc7f9"}
Sep 17 19:11:31.472164 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.472132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8" event={"ID":"a8dc49d1-dffd-4fc8-9920-a0173e96e689","Type":"ContainerStarted","Data":"0b39e4712f83c8b336793c717d523c65bcf5b3daed0dcede77b82feef92a25fc"}
Sep 17 19:11:31.472564 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.472225    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:31.481444 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.481419    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6dsls"]
Sep 17 19:11:31.483553 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.483534    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.486745 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.486729    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 19:11:31.486826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.486731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-dnhzz\""
Sep 17 19:11:31.486826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.486731    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 19:11:31.496322 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.496279    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8" podStartSLOduration=1.767988986 podStartE2EDuration="5.496261982s" totalImagesPullingTime="3.728272996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:31.4935828 +0000 UTC m=+293.541462657" watchObservedRunningTime="2026-09-17 19:11:31.496261982 +0000 UTC m=+293.544141840"
Sep 17 19:11:31.498871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.498853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6dsls"]
Sep 17 19:11:31.618253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.618225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2cv\" (UniqueName: \"kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-kube-api-access-pm2cv\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.618379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.618261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9cced643-5a9b-42ee-adaf-304dd1e39839-cabundle0\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.618379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.618285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-certificates\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.686453 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.686425    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"]
Sep 17 19:11:31.688642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.688625    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.691158 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.691140    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 19:11:31.698229 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.698209    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"]
Sep 17 19:11:31.718922 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.718892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-certificates\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.719024 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.718969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2cv\" (UniqueName: \"kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-kube-api-access-pm2cv\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.719024 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:31.718989    2583 secret.go:281] references non-existent secret key: ca.crt
Sep 17 19:11:31.719024 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:31.719009    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 19:11:31.719024 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:31.719020    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-6dsls: references non-existent secret key: ca.crt
Sep 17 19:11:31.719224 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:31.719072    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-certificates podName:9cced643-5a9b-42ee-adaf-304dd1e39839 nodeName:}" failed. No retries permitted until 2026-09-17 19:11:32.219053926 +0000 UTC m=+294.266933767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-certificates") pod "keda-operator-579c8958f7-6dsls" (UID: "9cced643-5a9b-42ee-adaf-304dd1e39839") : references non-existent secret key: ca.crt
Sep 17 19:11:31.719224 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.719010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9cced643-5a9b-42ee-adaf-304dd1e39839-cabundle0\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.719708 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.719668    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9cced643-5a9b-42ee-adaf-304dd1e39839-cabundle0\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.731892 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.731842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2cv\" (UniqueName: \"kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-kube-api-access-pm2cv\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:31.819915 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.819892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f9f48159-454e-49ef-9b5d-b18089dd4fd5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.820011 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.819950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f9f48159-454e-49ef-9b5d-b18089dd4fd5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.820063 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.820013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25hj\" (UniqueName: \"kubernetes.io/projected/f9f48159-454e-49ef-9b5d-b18089dd4fd5-kube-api-access-t25hj\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.905109 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.905081    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-sswwk"]
Sep 17 19:11:31.907616 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.907600    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:31.910375 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.910343    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 19:11:31.919055 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.919031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-sswwk"]
Sep 17 19:11:31.921364 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.921341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f9f48159-454e-49ef-9b5d-b18089dd4fd5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.921470 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.921400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f9f48159-454e-49ef-9b5d-b18089dd4fd5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.921527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.921492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25hj\" (UniqueName: \"kubernetes.io/projected/f9f48159-454e-49ef-9b5d-b18089dd4fd5-kube-api-access-t25hj\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.922118 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.922096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f9f48159-454e-49ef-9b5d-b18089dd4fd5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.924557 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.924535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f9f48159-454e-49ef-9b5d-b18089dd4fd5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.931233 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.931215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25hj\" (UniqueName: \"kubernetes.io/projected/f9f48159-454e-49ef-9b5d-b18089dd4fd5-kube-api-access-t25hj\") pod \"keda-metrics-apiserver-77857b7b98-992xf\" (UID: \"f9f48159-454e-49ef-9b5d-b18089dd4fd5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:31.999371 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:31.999307    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:32.022809 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.022787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fl6\" (UniqueName: \"kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-kube-api-access-49fl6\") pod \"keda-admission-6c4d864b86-sswwk\" (UID: \"86a92d09-1a19-451a-89c4-69bc344f7f32\") " pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:32.022912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.022820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-certificates\") pod \"keda-admission-6c4d864b86-sswwk\" (UID: \"86a92d09-1a19-451a-89c4-69bc344f7f32\") " pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:32.123860 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.123834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fl6\" (UniqueName: \"kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-kube-api-access-49fl6\") pod \"keda-admission-6c4d864b86-sswwk\" (UID: \"86a92d09-1a19-451a-89c4-69bc344f7f32\") " pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:32.124003 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.123869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-certificates\") pod \"keda-admission-6c4d864b86-sswwk\" (UID: \"86a92d09-1a19-451a-89c4-69bc344f7f32\") " pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:32.124048 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:32.124008    2583 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 19:11:32.124048 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:32.124025    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-sswwk: secret "keda-admission-webhooks-certs" not found
Sep 17 19:11:32.124111 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:11:32.124073    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-certificates podName:86a92d09-1a19-451a-89c4-69bc344f7f32 nodeName:}" failed. No retries permitted until 2026-09-17 19:11:32.624060032 +0000 UTC m=+294.671939867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-certificates") pod "keda-admission-6c4d864b86-sswwk" (UID: "86a92d09-1a19-451a-89c4-69bc344f7f32") : secret "keda-admission-webhooks-certs" not found
Sep 17 19:11:32.144090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.144066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fl6\" (UniqueName: \"kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-kube-api-access-49fl6\") pod \"keda-admission-6c4d864b86-sswwk\" (UID: \"86a92d09-1a19-451a-89c4-69bc344f7f32\") " pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:32.224474 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.224441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-certificates\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:32.226781 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.226762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9cced643-5a9b-42ee-adaf-304dd1e39839-certificates\") pod \"keda-operator-579c8958f7-6dsls\" (UID: \"9cced643-5a9b-42ee-adaf-304dd1e39839\") " pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:32.279917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.279897    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"]
Sep 17 19:11:32.282118 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:11:32.282096    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f48159_454e_49ef_9b5d_b18089dd4fd5.slice/crio-33747fd76a87ef33600836567790ee06c4b2c04f4f08ffd8063f3d5b6c31e36a WatchSource:0}: Error finding container 33747fd76a87ef33600836567790ee06c4b2c04f4f08ffd8063f3d5b6c31e36a: Status 404 returned error can't find the container with id 33747fd76a87ef33600836567790ee06c4b2c04f4f08ffd8063f3d5b6c31e36a
Sep 17 19:11:32.393062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.393039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:32.477539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.477513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf" event={"ID":"f9f48159-454e-49ef-9b5d-b18089dd4fd5","Type":"ContainerStarted","Data":"33747fd76a87ef33600836567790ee06c4b2c04f4f08ffd8063f3d5b6c31e36a"}
Sep 17 19:11:32.627950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.627922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-certificates\") pod \"keda-admission-6c4d864b86-sswwk\" (UID: \"86a92d09-1a19-451a-89c4-69bc344f7f32\") " pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:32.630348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.630330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/86a92d09-1a19-451a-89c4-69bc344f7f32-certificates\") pod \"keda-admission-6c4d864b86-sswwk\" (UID: \"86a92d09-1a19-451a-89c4-69bc344f7f32\") " pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:32.665239 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.665217    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-6dsls"]
Sep 17 19:11:32.667117 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:11:32.667093    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cced643_5a9b_42ee_adaf_304dd1e39839.slice/crio-b23020c4c48aeb4427923ffd77c1b11abd0b2e2d2e3545f77f625f5223ca197d WatchSource:0}: Error finding container b23020c4c48aeb4427923ffd77c1b11abd0b2e2d2e3545f77f625f5223ca197d: Status 404 returned error can't find the container with id b23020c4c48aeb4427923ffd77c1b11abd0b2e2d2e3545f77f625f5223ca197d
Sep 17 19:11:32.819614 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:32.819576    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:33.121268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:33.121235    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-sswwk"]
Sep 17 19:11:33.124376 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:11:33.124346    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a92d09_1a19_451a_89c4_69bc344f7f32.slice/crio-d4eeec21db453afb15555af878984614bf78e99e2bbffe23f753630c413e5122 WatchSource:0}: Error finding container d4eeec21db453afb15555af878984614bf78e99e2bbffe23f753630c413e5122: Status 404 returned error can't find the container with id d4eeec21db453afb15555af878984614bf78e99e2bbffe23f753630c413e5122
Sep 17 19:11:33.484713 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:33.484612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-sswwk" event={"ID":"86a92d09-1a19-451a-89c4-69bc344f7f32","Type":"ContainerStarted","Data":"d4eeec21db453afb15555af878984614bf78e99e2bbffe23f753630c413e5122"}
Sep 17 19:11:33.486274 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:33.486240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6dsls" event={"ID":"9cced643-5a9b-42ee-adaf-304dd1e39839","Type":"ContainerStarted","Data":"b23020c4c48aeb4427923ffd77c1b11abd0b2e2d2e3545f77f625f5223ca197d"}
Sep 17 19:11:35.500041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:35.500001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-sswwk" event={"ID":"86a92d09-1a19-451a-89c4-69bc344f7f32","Type":"ContainerStarted","Data":"263cb74108cd4a4e5720b068044d52895fbbcad9708844ac50e8890ce873c81f"}
Sep 17 19:11:35.500468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:35.500094    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:35.520541 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:35.520498    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-sswwk" podStartSLOduration=2.9078845380000002 podStartE2EDuration="4.520480578s" totalImagesPullingTime="1.61259604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:35.518069149 +0000 UTC m=+297.565949007" watchObservedRunningTime="2026-09-17 19:11:35.520480578 +0000 UTC m=+297.568360435"
Sep 17 19:11:37.511956 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:37.511922    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf" event={"ID":"f9f48159-454e-49ef-9b5d-b18089dd4fd5","Type":"ContainerStarted","Data":"df853b7a272a7963af653774a92904265f617b589d2232a3bcaf119aa2741075"}
Sep 17 19:11:37.512407 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:37.512017    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:37.513528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:37.513502    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-6dsls" event={"ID":"9cced643-5a9b-42ee-adaf-304dd1e39839","Type":"ContainerStarted","Data":"5024a65079170f05dcfe445c37454b4be3dd4e9c87ca40e9d59dc9048992f206"}
Sep 17 19:11:37.513648 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:37.513572    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:11:37.561912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:37.561862    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf" podStartSLOduration=1.451728609 podStartE2EDuration="6.561846558s" totalImagesPullingTime="5.110117949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:37.561362088 +0000 UTC m=+299.609241944" watchObservedRunningTime="2026-09-17 19:11:37.561846558 +0000 UTC m=+299.609726414"
Sep 17 19:11:37.600416 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:37.600365    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-6dsls" podStartSLOduration=1.874519941 podStartE2EDuration="6.600348506s" totalImagesPullingTime="4.725828565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:37.598828968 +0000 UTC m=+299.646708825" watchObservedRunningTime="2026-09-17 19:11:37.600348506 +0000 UTC m=+299.648228365"
Sep 17 19:11:40.128070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:40.128041    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 19:11:48.524261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:48.524235    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-992xf"
Sep 17 19:11:52.480271 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:52.480243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fnpz8"
Sep 17 19:11:56.508170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:56.508141    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-sswwk"
Sep 17 19:11:58.520915 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:11:58.520881    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-6dsls"
Sep 17 19:12:24.166007 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.165925    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"]
Sep 17 19:12:24.168267 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.168248    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.171071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.171046    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 19:12:24.171182 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.171169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pgp8f\""
Sep 17 19:12:24.172338 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.172305    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 19:12:24.180456 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.180436    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"]
Sep 17 19:12:24.215120 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.215097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/68cfae3e-d239-47d3-b6b2-8c502042ce33-kube-api-access-gc2vc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.215222 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.215150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.215222 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.215208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.315932 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.315903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/68cfae3e-d239-47d3-b6b2-8c502042ce33-kube-api-access-gc2vc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.316040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.315960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.316040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.315987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.316313 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.316298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.316349 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.316330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.324885 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.324861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/68cfae3e-d239-47d3-b6b2-8c502042ce33-kube-api-access-gc2vc\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.477034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.476966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:24.746618 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:24.746589    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"]
Sep 17 19:12:24.748773 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:12:24.748747    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cfae3e_d239_47d3_b6b2_8c502042ce33.slice/crio-298dae3d270c454fd86941c314dca65400edcf575f3e8413d39bca52cb452e83 WatchSource:0}: Error finding container 298dae3d270c454fd86941c314dca65400edcf575f3e8413d39bca52cb452e83: Status 404 returned error can't find the container with id 298dae3d270c454fd86941c314dca65400edcf575f3e8413d39bca52cb452e83
Sep 17 19:12:25.720948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:25.720867    2583 generic.go:356] "Generic (PLEG): container finished" podID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerID="a3be423235c7906940aec99b66a6ab070a5f55c3bbe7b22aabe6f2a747ab38e0" exitCode=0
Sep 17 19:12:25.720948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:25.720929    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2" event={"ID":"68cfae3e-d239-47d3-b6b2-8c502042ce33","Type":"ContainerDied","Data":"a3be423235c7906940aec99b66a6ab070a5f55c3bbe7b22aabe6f2a747ab38e0"}
Sep 17 19:12:25.721316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:25.720970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2" event={"ID":"68cfae3e-d239-47d3-b6b2-8c502042ce33","Type":"ContainerStarted","Data":"298dae3d270c454fd86941c314dca65400edcf575f3e8413d39bca52cb452e83"}
Sep 17 19:12:25.721898 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:25.721882    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:12:28.736077 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:28.736041    2583 generic.go:356] "Generic (PLEG): container finished" podID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerID="30697549ee4246452aac422a64caa593420422f2c1313a9fbb374a64003a7746" exitCode=0
Sep 17 19:12:28.736439 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:28.736114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2" event={"ID":"68cfae3e-d239-47d3-b6b2-8c502042ce33","Type":"ContainerDied","Data":"30697549ee4246452aac422a64caa593420422f2c1313a9fbb374a64003a7746"}
Sep 17 19:12:29.742747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:29.742655    2583 generic.go:356] "Generic (PLEG): container finished" podID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerID="dc469867990b7f3d53236547cd0df78d069535f827d0d2eec14632462d738772" exitCode=0
Sep 17 19:12:29.743142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:29.742742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2" event={"ID":"68cfae3e-d239-47d3-b6b2-8c502042ce33","Type":"ContainerDied","Data":"dc469867990b7f3d53236547cd0df78d069535f827d0d2eec14632462d738772"}
Sep 17 19:12:30.863711 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:30.863662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:30.964435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:30.964402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/68cfae3e-d239-47d3-b6b2-8c502042ce33-kube-api-access-gc2vc\") pod \"68cfae3e-d239-47d3-b6b2-8c502042ce33\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") "
Sep 17 19:12:30.964435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:30.964442    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-bundle\") pod \"68cfae3e-d239-47d3-b6b2-8c502042ce33\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") "
Sep 17 19:12:30.964645 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:30.964477    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-util\") pod \"68cfae3e-d239-47d3-b6b2-8c502042ce33\" (UID: \"68cfae3e-d239-47d3-b6b2-8c502042ce33\") "
Sep 17 19:12:30.965189 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:30.965164    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-bundle" (OuterVolumeSpecName: "bundle") pod "68cfae3e-d239-47d3-b6b2-8c502042ce33" (UID: "68cfae3e-d239-47d3-b6b2-8c502042ce33"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:30.966761 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:30.966734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cfae3e-d239-47d3-b6b2-8c502042ce33-kube-api-access-gc2vc" (OuterVolumeSpecName: "kube-api-access-gc2vc") pod "68cfae3e-d239-47d3-b6b2-8c502042ce33" (UID: "68cfae3e-d239-47d3-b6b2-8c502042ce33"). InnerVolumeSpecName "kube-api-access-gc2vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:12:30.969443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:30.969421    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-util" (OuterVolumeSpecName: "util") pod "68cfae3e-d239-47d3-b6b2-8c502042ce33" (UID: "68cfae3e-d239-47d3-b6b2-8c502042ce33"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:31.065336 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:31.065308    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-util\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:12:31.065336 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:31.065336    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2vc\" (UniqueName: \"kubernetes.io/projected/68cfae3e-d239-47d3-b6b2-8c502042ce33-kube-api-access-gc2vc\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:12:31.065486 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:31.065345    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68cfae3e-d239-47d3-b6b2-8c502042ce33-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:12:31.753377 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:31.753345    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2"
Sep 17 19:12:31.753377 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:31.753362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409vfm2" event={"ID":"68cfae3e-d239-47d3-b6b2-8c502042ce33","Type":"ContainerDied","Data":"298dae3d270c454fd86941c314dca65400edcf575f3e8413d39bca52cb452e83"}
Sep 17 19:12:31.753615 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:31.753407    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298dae3d270c454fd86941c314dca65400edcf575f3e8413d39bca52cb452e83"
Sep 17 19:12:37.262539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.262500    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"]
Sep 17 19:12:37.263018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.262960    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerName="pull"
Sep 17 19:12:37.263018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.262977    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerName="pull"
Sep 17 19:12:37.263018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.262991    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerName="util"
Sep 17 19:12:37.263018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.262998    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerName="util"
Sep 17 19:12:37.263018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.263010    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerName="extract"
Sep 17 19:12:37.263018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.263018    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerName="extract"
Sep 17 19:12:37.263303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.263117    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="68cfae3e-d239-47d3-b6b2-8c502042ce33" containerName="extract"
Sep 17 19:12:37.265184 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.265164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.268046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.268027    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:12:37.268702 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.268671    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mb8mb\""
Sep 17 19:12:37.268775 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.268708    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 19:12:37.280709 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.280669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"]
Sep 17 19:12:37.315185 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.315165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtt5t\" (UniqueName: \"kubernetes.io/projected/8a92063d-e174-4cd7-bc35-28c58aa0ebdf-kube-api-access-vtt5t\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tctpv\" (UID: \"8a92063d-e174-4cd7-bc35-28c58aa0ebdf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.315279 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.315202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a92063d-e174-4cd7-bc35-28c58aa0ebdf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tctpv\" (UID: \"8a92063d-e174-4cd7-bc35-28c58aa0ebdf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.416165 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.416140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtt5t\" (UniqueName: \"kubernetes.io/projected/8a92063d-e174-4cd7-bc35-28c58aa0ebdf-kube-api-access-vtt5t\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tctpv\" (UID: \"8a92063d-e174-4cd7-bc35-28c58aa0ebdf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.416252 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.416183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a92063d-e174-4cd7-bc35-28c58aa0ebdf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tctpv\" (UID: \"8a92063d-e174-4cd7-bc35-28c58aa0ebdf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.416502 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.416486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a92063d-e174-4cd7-bc35-28c58aa0ebdf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tctpv\" (UID: \"8a92063d-e174-4cd7-bc35-28c58aa0ebdf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.429188 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.429166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtt5t\" (UniqueName: \"kubernetes.io/projected/8a92063d-e174-4cd7-bc35-28c58aa0ebdf-kube-api-access-vtt5t\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tctpv\" (UID: \"8a92063d-e174-4cd7-bc35-28c58aa0ebdf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.574000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.573983    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"
Sep 17 19:12:37.857422 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:37.857366    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv"]
Sep 17 19:12:37.860482 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:12:37.860458    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a92063d_e174_4cd7_bc35_28c58aa0ebdf.slice/crio-ccfb5e1366862737e5b0e002a7517e7171c6f14db5d45d898105ba7dc797bef6 WatchSource:0}: Error finding container ccfb5e1366862737e5b0e002a7517e7171c6f14db5d45d898105ba7dc797bef6: Status 404 returned error can't find the container with id ccfb5e1366862737e5b0e002a7517e7171c6f14db5d45d898105ba7dc797bef6
Sep 17 19:12:38.790403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:38.790362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv" event={"ID":"8a92063d-e174-4cd7-bc35-28c58aa0ebdf","Type":"ContainerStarted","Data":"ccfb5e1366862737e5b0e002a7517e7171c6f14db5d45d898105ba7dc797bef6"}
Sep 17 19:12:39.797220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:39.797190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv" event={"ID":"8a92063d-e174-4cd7-bc35-28c58aa0ebdf","Type":"ContainerStarted","Data":"056256638a01d7c01fc7a5535f66526b2a808ce5fabf62cff9a4e99b8d1d67bf"}
Sep 17 19:12:39.831149 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:39.831102    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tctpv" podStartSLOduration=1.245131935 podStartE2EDuration="2.831087174s" totalImagesPullingTime="1.585955239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:39.828535268 +0000 UTC m=+361.876415124" watchObservedRunningTime="2026-09-17 19:12:39.831087174 +0000 UTC m=+361.878967031"
Sep 17 19:12:46.313668 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.313633    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"]
Sep 17 19:12:46.316159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.316141    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.319195 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.319172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pgp8f\""
Sep 17 19:12:46.319302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.319181    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 19:12:46.319302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.319182    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 19:12:46.325875 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.325855    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"]
Sep 17 19:12:46.393435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.393411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.393514 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.393448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrl9\" (UniqueName: \"kubernetes.io/projected/124a3ecd-84ee-4332-99ce-06f2e367dae4-kube-api-access-cdrl9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.393514 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.393488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.494292 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.494268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.494388 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.494304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrl9\" (UniqueName: \"kubernetes.io/projected/124a3ecd-84ee-4332-99ce-06f2e367dae4-kube-api-access-cdrl9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.494388 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.494330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.494636 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.494614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.494709 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.494665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.504767 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.504738    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrl9\" (UniqueName: \"kubernetes.io/projected/124a3ecd-84ee-4332-99ce-06f2e367dae4-kube-api-access-cdrl9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.625450 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.625398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:46.909397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:46.909327    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"]
Sep 17 19:12:46.912418 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:12:46.912392    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124a3ecd_84ee_4332_99ce_06f2e367dae4.slice/crio-ffb10d70e7b0c38a797ce35bb44d97858c9586b7102b42aba013bc8f0d3c65c3 WatchSource:0}: Error finding container ffb10d70e7b0c38a797ce35bb44d97858c9586b7102b42aba013bc8f0d3c65c3: Status 404 returned error can't find the container with id ffb10d70e7b0c38a797ce35bb44d97858c9586b7102b42aba013bc8f0d3c65c3
Sep 17 19:12:47.834060 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:47.834027    2583 generic.go:356] "Generic (PLEG): container finished" podID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerID="4dc497a04ea6e3e684c14bd7370d4dbe75159f0a755af09520e6890754e0a7db" exitCode=0
Sep 17 19:12:47.834507 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:47.834121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6" event={"ID":"124a3ecd-84ee-4332-99ce-06f2e367dae4","Type":"ContainerDied","Data":"4dc497a04ea6e3e684c14bd7370d4dbe75159f0a755af09520e6890754e0a7db"}
Sep 17 19:12:47.834507 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:47.834200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6" event={"ID":"124a3ecd-84ee-4332-99ce-06f2e367dae4","Type":"ContainerStarted","Data":"ffb10d70e7b0c38a797ce35bb44d97858c9586b7102b42aba013bc8f0d3c65c3"}
Sep 17 19:12:48.229257 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.229189    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-645bf46b68-88fhx"]
Sep 17 19:12:48.231286 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.231271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.254906 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.254881    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645bf46b68-88fhx"]
Sep 17 19:12:48.309707 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.309646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-trusted-ca-bundle\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.309849 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.309719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-service-ca\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.309849 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.309738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hbk\" (UniqueName: \"kubernetes.io/projected/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-kube-api-access-h7hbk\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.309849 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.309795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-config\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.309849 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.309841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-oauth-serving-cert\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.309995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.309870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-oauth-config\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.309995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.309902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-serving-cert\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.411211 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.411180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-trusted-ca-bundle\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.411361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.411234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-service-ca\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.411361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.411259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hbk\" (UniqueName: \"kubernetes.io/projected/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-kube-api-access-h7hbk\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.411361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.411312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-config\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.411520 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.411389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-oauth-serving-cert\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.411520 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.411434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-oauth-config\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.411520 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.411477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-serving-cert\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.412169 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.412100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-service-ca\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.412169 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.412108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-trusted-ca-bundle\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.412169 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.412154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-oauth-serving-cert\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.412169 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.412167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-config\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.414398 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.414376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-serving-cert\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.414904 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.414858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-console-oauth-config\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.420094 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.420074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hbk\" (UniqueName: \"kubernetes.io/projected/7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447-kube-api-access-h7hbk\") pod \"console-645bf46b68-88fhx\" (UID: \"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447\") " pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.540274 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.540187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:48.822743 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.822715    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645bf46b68-88fhx"]
Sep 17 19:12:48.824737 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:12:48.824707    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbc1c4a_b3e2_42ca_b5aa_0d5107fbe447.slice/crio-359d145367749a9b63ebd1ab1b4ebc1d62302d9cbf7643a482976200b9bff3c3 WatchSource:0}: Error finding container 359d145367749a9b63ebd1ab1b4ebc1d62302d9cbf7643a482976200b9bff3c3: Status 404 returned error can't find the container with id 359d145367749a9b63ebd1ab1b4ebc1d62302d9cbf7643a482976200b9bff3c3
Sep 17 19:12:48.842196 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:48.842167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645bf46b68-88fhx" event={"ID":"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447","Type":"ContainerStarted","Data":"359d145367749a9b63ebd1ab1b4ebc1d62302d9cbf7643a482976200b9bff3c3"}
Sep 17 19:12:49.851436 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:49.851399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645bf46b68-88fhx" event={"ID":"7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447","Type":"ContainerStarted","Data":"4d3808e3ee590eceef5912d78595d9836cda5fd0c11e98f991a39a572d8df36f"}
Sep 17 19:12:49.853240 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:49.853215    2583 generic.go:356] "Generic (PLEG): container finished" podID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerID="93f4c79ee7bdd1960195d99b6e9112824f2198a7519a99539a7f149e8fb335c5" exitCode=0
Sep 17 19:12:49.853346 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:49.853275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6" event={"ID":"124a3ecd-84ee-4332-99ce-06f2e367dae4","Type":"ContainerDied","Data":"93f4c79ee7bdd1960195d99b6e9112824f2198a7519a99539a7f149e8fb335c5"}
Sep 17 19:12:49.871496 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:49.871443    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-645bf46b68-88fhx" podStartSLOduration=1.8714214949999999 podStartE2EDuration="1.871421495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:49.869356393 +0000 UTC m=+371.917236251" watchObservedRunningTime="2026-09-17 19:12:49.871421495 +0000 UTC m=+371.919301353"
Sep 17 19:12:50.859782 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:50.859739    2583 generic.go:356] "Generic (PLEG): container finished" podID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerID="6e164cc1475f97c1a951f8807966d17f89de79f8ebabc55488ec4fcb999d1e07" exitCode=0
Sep 17 19:12:50.860138 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:50.859813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6" event={"ID":"124a3ecd-84ee-4332-99ce-06f2e367dae4","Type":"ContainerDied","Data":"6e164cc1475f97c1a951f8807966d17f89de79f8ebabc55488ec4fcb999d1e07"}
Sep 17 19:12:51.990605 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:51.990580    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:52.045101 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.045076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-bundle\") pod \"124a3ecd-84ee-4332-99ce-06f2e367dae4\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") "
Sep 17 19:12:52.045226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.045111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-util\") pod \"124a3ecd-84ee-4332-99ce-06f2e367dae4\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") "
Sep 17 19:12:52.045226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.045198    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdrl9\" (UniqueName: \"kubernetes.io/projected/124a3ecd-84ee-4332-99ce-06f2e367dae4-kube-api-access-cdrl9\") pod \"124a3ecd-84ee-4332-99ce-06f2e367dae4\" (UID: \"124a3ecd-84ee-4332-99ce-06f2e367dae4\") "
Sep 17 19:12:52.045548 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.045510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-bundle" (OuterVolumeSpecName: "bundle") pod "124a3ecd-84ee-4332-99ce-06f2e367dae4" (UID: "124a3ecd-84ee-4332-99ce-06f2e367dae4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:52.047356 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.047332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124a3ecd-84ee-4332-99ce-06f2e367dae4-kube-api-access-cdrl9" (OuterVolumeSpecName: "kube-api-access-cdrl9") pod "124a3ecd-84ee-4332-99ce-06f2e367dae4" (UID: "124a3ecd-84ee-4332-99ce-06f2e367dae4"). InnerVolumeSpecName "kube-api-access-cdrl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:12:52.050546 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.050528    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-util" (OuterVolumeSpecName: "util") pod "124a3ecd-84ee-4332-99ce-06f2e367dae4" (UID: "124a3ecd-84ee-4332-99ce-06f2e367dae4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:52.146491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.146434    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:12:52.146491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.146454    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/124a3ecd-84ee-4332-99ce-06f2e367dae4-util\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:12:52.146491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.146463    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrl9\" (UniqueName: \"kubernetes.io/projected/124a3ecd-84ee-4332-99ce-06f2e367dae4-kube-api-access-cdrl9\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:12:52.871523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.871492    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6"
Sep 17 19:12:52.871523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.871506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f965v6" event={"ID":"124a3ecd-84ee-4332-99ce-06f2e367dae4","Type":"ContainerDied","Data":"ffb10d70e7b0c38a797ce35bb44d97858c9586b7102b42aba013bc8f0d3c65c3"}
Sep 17 19:12:52.871751 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:52.871550    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb10d70e7b0c38a797ce35bb44d97858c9586b7102b42aba013bc8f0d3c65c3"
Sep 17 19:12:56.623618 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.623583    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"]
Sep 17 19:12:56.624040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.623923    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerName="extract"
Sep 17 19:12:56.624040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.623935    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerName="extract"
Sep 17 19:12:56.624040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.623949    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerName="util"
Sep 17 19:12:56.624040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.623954    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerName="util"
Sep 17 19:12:56.624040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.623961    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerName="pull"
Sep 17 19:12:56.624040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.623966    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerName="pull"
Sep 17 19:12:56.624040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.624022    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="124a3ecd-84ee-4332-99ce-06f2e367dae4" containerName="extract"
Sep 17 19:12:56.626175 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.626160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:56.629078 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.629056    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 19:12:56.629926 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.629910    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7pvbp\""
Sep 17 19:12:56.629980 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.629911    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:12:56.636212 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.636189    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"]
Sep 17 19:12:56.680637 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.680611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e06d5ff7-b02f-4706-84c6-ed9229ae262a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gnnrq\" (UID: \"e06d5ff7-b02f-4706-84c6-ed9229ae262a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:56.680775 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.680662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8cx\" (UniqueName: \"kubernetes.io/projected/e06d5ff7-b02f-4706-84c6-ed9229ae262a-kube-api-access-cw8cx\") pod \"openshift-lws-operator-fd8ccff4c-gnnrq\" (UID: \"e06d5ff7-b02f-4706-84c6-ed9229ae262a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:56.782036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.782008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e06d5ff7-b02f-4706-84c6-ed9229ae262a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gnnrq\" (UID: \"e06d5ff7-b02f-4706-84c6-ed9229ae262a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:56.782162 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.782058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8cx\" (UniqueName: \"kubernetes.io/projected/e06d5ff7-b02f-4706-84c6-ed9229ae262a-kube-api-access-cw8cx\") pod \"openshift-lws-operator-fd8ccff4c-gnnrq\" (UID: \"e06d5ff7-b02f-4706-84c6-ed9229ae262a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:56.782377 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.782359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e06d5ff7-b02f-4706-84c6-ed9229ae262a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gnnrq\" (UID: \"e06d5ff7-b02f-4706-84c6-ed9229ae262a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:56.790836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.790815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8cx\" (UniqueName: \"kubernetes.io/projected/e06d5ff7-b02f-4706-84c6-ed9229ae262a-kube-api-access-cw8cx\") pod \"openshift-lws-operator-fd8ccff4c-gnnrq\" (UID: \"e06d5ff7-b02f-4706-84c6-ed9229ae262a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:56.943354 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:56.943288    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"
Sep 17 19:12:57.213194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:57.213169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq"]
Sep 17 19:12:57.215439 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:12:57.215410    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode06d5ff7_b02f_4706_84c6_ed9229ae262a.slice/crio-ec6c36dca97e9b5094a358ef153e2c799ab1e01eb6a18cffdb00c6f2ef0fb5b5 WatchSource:0}: Error finding container ec6c36dca97e9b5094a358ef153e2c799ab1e01eb6a18cffdb00c6f2ef0fb5b5: Status 404 returned error can't find the container with id ec6c36dca97e9b5094a358ef153e2c799ab1e01eb6a18cffdb00c6f2ef0fb5b5
Sep 17 19:12:57.902353 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:57.902319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq" event={"ID":"e06d5ff7-b02f-4706-84c6-ed9229ae262a","Type":"ContainerStarted","Data":"ec6c36dca97e9b5094a358ef153e2c799ab1e01eb6a18cffdb00c6f2ef0fb5b5"}
Sep 17 19:12:58.540384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:58.540351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:58.540548 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:58.540415    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:58.545243 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:58.545217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:58.910483 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:58.910451    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645bf46b68-88fhx"
Sep 17 19:12:58.960712 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:12:58.960652    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4585dc68-dxqpz"]
Sep 17 19:13:00.920647 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:00.920610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq" event={"ID":"e06d5ff7-b02f-4706-84c6-ed9229ae262a","Type":"ContainerStarted","Data":"087dd752baface352e55fbe8042564902ed953df99db512aa9364fb37a755351"}
Sep 17 19:13:00.944119 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:00.944067    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnnrq" podStartSLOduration=2.166156759 podStartE2EDuration="4.944053374s" totalImagesPullingTime="2.777896615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:00.942475853 +0000 UTC m=+382.990355711" watchObservedRunningTime="2026-09-17 19:13:00.944053374 +0000 UTC m=+382.991933231"
Sep 17 19:13:08.563131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.563096    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"]
Sep 17 19:13:08.573131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.573098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.576208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.576164    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 19:13:08.576396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.576209    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 19:13:08.576396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.576214    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 19:13:08.576551 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.576415    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:13:08.576551 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.576501    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 19:13:08.576824 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.576807    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 19:13:08.577266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.577253    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wb887\""
Sep 17 19:13:08.583165 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.583143    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"]
Sep 17 19:13:08.685899 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.685863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.685899 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.685899    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.686123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.685938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.686123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.685979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.686123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.686000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3168d53-2398-4744-99ae-4f5b21c10024-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.686123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.686023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.686123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.686050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4rq\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-kube-api-access-hb4rq\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.786478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.786440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.786478 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.786478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.786724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.786531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.786724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.786561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.786724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.786589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3168d53-2398-4744-99ae-4f5b21c10024-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.786724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.786627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.786724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.786668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4rq\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-kube-api-access-hb4rq\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.787452 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.787328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.788979 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.788955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3168d53-2398-4744-99ae-4f5b21c10024-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.789110 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.789091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.789238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.789221    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.789300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.789279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.795675 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.795647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4rq\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-kube-api-access-hb4rq\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.795675 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.795670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lq7q7\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:08.883899 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:08.883814    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:09.159598 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:09.159575    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"]
Sep 17 19:13:09.161961 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:09.161936    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3168d53_2398_4744_99ae_4f5b21c10024.slice/crio-2fd4f3a81505ec62a5e2f08ec8f92392164594fd5eb349710447a84d1dd25861 WatchSource:0}: Error finding container 2fd4f3a81505ec62a5e2f08ec8f92392164594fd5eb349710447a84d1dd25861: Status 404 returned error can't find the container with id 2fd4f3a81505ec62a5e2f08ec8f92392164594fd5eb349710447a84d1dd25861
Sep 17 19:13:09.973010 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:09.972972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7" event={"ID":"d3168d53-2398-4744-99ae-4f5b21c10024","Type":"ContainerStarted","Data":"2fd4f3a81505ec62a5e2f08ec8f92392164594fd5eb349710447a84d1dd25861"}
Sep 17 19:13:12.747898 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:12.747857    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 17 19:13:12.748178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:12.747928    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 17 19:13:12.989607 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:12.989578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7" event={"ID":"d3168d53-2398-4744-99ae-4f5b21c10024","Type":"ContainerStarted","Data":"71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61"}
Sep 17 19:13:12.989759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:12.989720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:13.013329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:13.013251    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7" podStartSLOduration=1.4296930030000001 podStartE2EDuration="5.013233335s" totalImagesPullingTime="3.583540332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:13.012277098 +0000 UTC m=+395.060156955" watchObservedRunningTime="2026-09-17 19:13:13.013233335 +0000 UTC m=+395.061113193"
Sep 17 19:13:13.996374 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:13.996327    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-lq7q7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 19:13:13.996790 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:13.996396    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7" podUID="d3168d53-2398-4744-99ae-4f5b21c10024" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:16.995655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:16.995618    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:13:23.983004 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:23.982958    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f4585dc68-dxqpz" podUID="bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" containerName="console" containerID="cri-o://48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28" gracePeriod=15
Sep 17 19:13:24.225668 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.225645    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:13:24.321126 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-config\") pod \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") "
Sep 17 19:13:24.321266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5hbd\" (UniqueName: \"kubernetes.io/projected/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-kube-api-access-k5hbd\") pod \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") "
Sep 17 19:13:24.321266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321188    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-serving-cert\") pod \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") "
Sep 17 19:13:24.321266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321210    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-oauth-serving-cert\") pod \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") "
Sep 17 19:13:24.321424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-oauth-config\") pod \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") "
Sep 17 19:13:24.321424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321309    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-service-ca\") pod \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") "
Sep 17 19:13:24.321424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321353    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-trusted-ca-bundle\") pod \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\" (UID: \"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2\") "
Sep 17 19:13:24.321582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321426    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-config" (OuterVolumeSpecName: "console-config") pod "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" (UID: "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:24.321642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321615    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:24.321782 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321752    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" (UID: "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:24.322094 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321862    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" (UID: "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:24.322094 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.321910    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-service-ca" (OuterVolumeSpecName: "service-ca") pod "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" (UID: "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:24.323608 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.323582    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" (UID: "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:24.323722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.323611    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" (UID: "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:24.323722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.323641    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-kube-api-access-k5hbd" (OuterVolumeSpecName: "kube-api-access-k5hbd") pod "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" (UID: "bde25ad3-adfe-4db0-8bae-af4c2f6d49a2"). InnerVolumeSpecName "kube-api-access-k5hbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:24.422396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.422352    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5hbd\" (UniqueName: \"kubernetes.io/projected/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-kube-api-access-k5hbd\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:24.422396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.422387    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:24.422396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.422396    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-oauth-serving-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:24.422396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.422404    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-console-oauth-config\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:24.422396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.422413    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-service-ca\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:24.422721 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:24.422421    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2-trusted-ca-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:25.046495 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.046452    2583 generic.go:356] "Generic (PLEG): container finished" podID="bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" containerID="48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28" exitCode=0
Sep 17 19:13:25.046913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.046519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4585dc68-dxqpz" event={"ID":"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2","Type":"ContainerDied","Data":"48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28"}
Sep 17 19:13:25.046913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.046557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4585dc68-dxqpz" event={"ID":"bde25ad3-adfe-4db0-8bae-af4c2f6d49a2","Type":"ContainerDied","Data":"99da12e90554864c2545a0eed959b59893d654c47b66bc289e41f4f4b5d7bd73"}
Sep 17 19:13:25.046913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.046576    2583 scope.go:117] "RemoveContainer" containerID="48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28"
Sep 17 19:13:25.046913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.046520    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4585dc68-dxqpz"
Sep 17 19:13:25.056059 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.056041    2583 scope.go:117] "RemoveContainer" containerID="48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28"
Sep 17 19:13:25.056338 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:13:25.056322    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28\": container with ID starting with 48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28 not found: ID does not exist" containerID="48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28"
Sep 17 19:13:25.056380 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.056346    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28"} err="failed to get container status \"48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28\": rpc error: code = NotFound desc = could not find container \"48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28\": container with ID starting with 48fe896592e0f82cd8aa42df0e8ca5883bd4054ce27a4176d05f48f6412ddd28 not found: ID does not exist"
Sep 17 19:13:25.068669 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.068643    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4585dc68-dxqpz"]
Sep 17 19:13:25.073361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:25.073337    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f4585dc68-dxqpz"]
Sep 17 19:13:26.255351 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.255319    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" path="/var/lib/kubelet/pods/bde25ad3-adfe-4db0-8bae-af4c2f6d49a2/volumes"
Sep 17 19:13:26.813435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.813401    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"]
Sep 17 19:13:26.813764 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.813751    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" containerName="console"
Sep 17 19:13:26.813815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.813766    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" containerName="console"
Sep 17 19:13:26.813851 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.813815    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bde25ad3-adfe-4db0-8bae-af4c2f6d49a2" containerName="console"
Sep 17 19:13:26.818759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.818738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.821766 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.821747    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 19:13:26.821875 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.821789    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 19:13:26.822911 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.822886    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pgp8f\""
Sep 17 19:13:26.828538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.828511    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"]
Sep 17 19:13:26.837969 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.837945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.838063 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.838002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.838104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.838088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjbp\" (UniqueName: \"kubernetes.io/projected/2ffac514-fcbd-4177-bc22-ca4f585f55bf-kube-api-access-lpjbp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.938520 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.938494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjbp\" (UniqueName: \"kubernetes.io/projected/2ffac514-fcbd-4177-bc22-ca4f585f55bf-kube-api-access-lpjbp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.938623 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.938548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.938623 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.938573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.938893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.938875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.938958 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.938911    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.968058 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.968037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjbp\" (UniqueName: \"kubernetes.io/projected/2ffac514-fcbd-4177-bc22-ca4f585f55bf-kube-api-access-lpjbp\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:26.980790 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.980769    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"]
Sep 17 19:13:26.984221 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.984197    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:26.995955 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:26.995936    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"]
Sep 17 19:13:27.038942 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.038918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.039028 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.038956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.039028 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.038984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lw4j\" (UniqueName: \"kubernetes.io/projected/b6a24697-6a01-4808-acee-132cff3f8208-kube-api-access-2lw4j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.049880 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.049855    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"]
Sep 17 19:13:27.053177 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.053161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.062431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.062413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"]
Sep 17 19:13:27.128426 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.128363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:27.139556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.139534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.139656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.139577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.139656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.139607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.139767 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.139712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.139767 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.139758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zs2k\" (UniqueName: \"kubernetes.io/projected/f9888543-821e-4c89-a6cd-216c34f1bd8a-kube-api-access-5zs2k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.139988 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.139791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lw4j\" (UniqueName: \"kubernetes.io/projected/b6a24697-6a01-4808-acee-132cff3f8208-kube-api-access-2lw4j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.139988 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.139963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.140097 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.140063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.146155 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.146133    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"]
Sep 17 19:13:27.148813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.148792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lw4j\" (UniqueName: \"kubernetes.io/projected/b6a24697-6a01-4808-acee-132cff3f8208-kube-api-access-2lw4j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.150915 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.150898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.158621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.158484    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"]
Sep 17 19:13:27.240412 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zs2k\" (UniqueName: \"kubernetes.io/projected/f9888543-821e-4c89-a6cd-216c34f1bd8a-kube-api-access-5zs2k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.240557 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hfx\" (UniqueName: \"kubernetes.io/projected/a33db913-5dc8-4533-a6de-7b071065a909-kube-api-access-74hfx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.240557 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.240557 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.240732 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.240732 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.240897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.240962 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.240937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.259032 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.259003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zs2k\" (UniqueName: \"kubernetes.io/projected/f9888543-821e-4c89-a6cd-216c34f1bd8a-kube-api-access-5zs2k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.293444 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.293419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:27.342154 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.342117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.342279 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.342219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.342279 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.342271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hfx\" (UniqueName: \"kubernetes.io/projected/a33db913-5dc8-4533-a6de-7b071065a909-kube-api-access-74hfx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.342567 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.342543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.342738 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.342721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.351247 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.351221    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hfx\" (UniqueName: \"kubernetes.io/projected/a33db913-5dc8-4533-a6de-7b071065a909-kube-api-access-74hfx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.362040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.362015    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:27.405558 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.405529    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"]
Sep 17 19:13:27.407148 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:27.407120    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffac514_fcbd_4177_bc22_ca4f585f55bf.slice/crio-e175a9eb38a86a15ed3feedae65c9780bac89af1d12d44b3db046ae7f153127f WatchSource:0}: Error finding container e175a9eb38a86a15ed3feedae65c9780bac89af1d12d44b3db046ae7f153127f: Status 404 returned error can't find the container with id e175a9eb38a86a15ed3feedae65c9780bac89af1d12d44b3db046ae7f153127f
Sep 17 19:13:27.465824 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.465802    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:27.563458 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.563413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"]
Sep 17 19:13:27.566096 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:27.566074    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a24697_6a01_4808_acee_132cff3f8208.slice/crio-56727226cb6db70ccc50edb255903c81088ffe952b7de1cfdec4f61c09aca2a0 WatchSource:0}: Error finding container 56727226cb6db70ccc50edb255903c81088ffe952b7de1cfdec4f61c09aca2a0: Status 404 returned error can't find the container with id 56727226cb6db70ccc50edb255903c81088ffe952b7de1cfdec4f61c09aca2a0
Sep 17 19:13:27.638465 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.638438    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"]
Sep 17 19:13:27.641990 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:27.641967    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9888543_821e_4c89_a6cd_216c34f1bd8a.slice/crio-57f0acd6485c5d6544bbd243976515867452d589c56d904c6e036ebaffb726ab WatchSource:0}: Error finding container 57f0acd6485c5d6544bbd243976515867452d589c56d904c6e036ebaffb726ab: Status 404 returned error can't find the container with id 57f0acd6485c5d6544bbd243976515867452d589c56d904c6e036ebaffb726ab
Sep 17 19:13:27.751507 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:27.751479    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"]
Sep 17 19:13:27.758931 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:27.758904    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33db913_5dc8_4533_a6de_7b071065a909.slice/crio-538ed0f00a085c726c51f9a1772c7f20562d79221304af233ae1ee1f2b5326fb WatchSource:0}: Error finding container 538ed0f00a085c726c51f9a1772c7f20562d79221304af233ae1ee1f2b5326fb: Status 404 returned error can't find the container with id 538ed0f00a085c726c51f9a1772c7f20562d79221304af233ae1ee1f2b5326fb
Sep 17 19:13:28.063699 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.063652    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerID="7760b2574bfc278cb189a5cdfce7729c1343091e2cadfb4f917450d7f7e1ece8" exitCode=0
Sep 17 19:13:28.063820 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.063728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg" event={"ID":"2ffac514-fcbd-4177-bc22-ca4f585f55bf","Type":"ContainerDied","Data":"7760b2574bfc278cb189a5cdfce7729c1343091e2cadfb4f917450d7f7e1ece8"}
Sep 17 19:13:28.063820 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.063782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg" event={"ID":"2ffac514-fcbd-4177-bc22-ca4f585f55bf","Type":"ContainerStarted","Data":"e175a9eb38a86a15ed3feedae65c9780bac89af1d12d44b3db046ae7f153127f"}
Sep 17 19:13:28.065444 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.065418    2583 generic.go:356] "Generic (PLEG): container finished" podID="a33db913-5dc8-4533-a6de-7b071065a909" containerID="4aa53081f5215dcb17a1ee29cb10dc8143c357b1af8178d8d7916661ead6e2f6" exitCode=0
Sep 17 19:13:28.065529 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.065496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5" event={"ID":"a33db913-5dc8-4533-a6de-7b071065a909","Type":"ContainerDied","Data":"4aa53081f5215dcb17a1ee29cb10dc8143c357b1af8178d8d7916661ead6e2f6"}
Sep 17 19:13:28.065593 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.065541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5" event={"ID":"a33db913-5dc8-4533-a6de-7b071065a909","Type":"ContainerStarted","Data":"538ed0f00a085c726c51f9a1772c7f20562d79221304af233ae1ee1f2b5326fb"}
Sep 17 19:13:28.067253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.067232    2583 generic.go:356] "Generic (PLEG): container finished" podID="b6a24697-6a01-4808-acee-132cff3f8208" containerID="138f0d9ba23148d809aca045045ed8db94a1978de5d1cffc795f460492fcd018" exitCode=0
Sep 17 19:13:28.067369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.067296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69" event={"ID":"b6a24697-6a01-4808-acee-132cff3f8208","Type":"ContainerDied","Data":"138f0d9ba23148d809aca045045ed8db94a1978de5d1cffc795f460492fcd018"}
Sep 17 19:13:28.067369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.067344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69" event={"ID":"b6a24697-6a01-4808-acee-132cff3f8208","Type":"ContainerStarted","Data":"56727226cb6db70ccc50edb255903c81088ffe952b7de1cfdec4f61c09aca2a0"}
Sep 17 19:13:28.069207 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.069014    2583 generic.go:356] "Generic (PLEG): container finished" podID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerID="e871f88fa72f1b74cff1a837d3355c6dcccb8869ae3e64bbc3640c75dfe027ca" exitCode=0
Sep 17 19:13:28.069207 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.069048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq" event={"ID":"f9888543-821e-4c89-a6cd-216c34f1bd8a","Type":"ContainerDied","Data":"e871f88fa72f1b74cff1a837d3355c6dcccb8869ae3e64bbc3640c75dfe027ca"}
Sep 17 19:13:28.069207 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:28.069102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq" event={"ID":"f9888543-821e-4c89-a6cd-216c34f1bd8a","Type":"ContainerStarted","Data":"57f0acd6485c5d6544bbd243976515867452d589c56d904c6e036ebaffb726ab"}
Sep 17 19:13:29.077705 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:29.077664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq" event={"ID":"f9888543-821e-4c89-a6cd-216c34f1bd8a","Type":"ContainerStarted","Data":"ddb6793e2884f3ebb16ea163851be8d1a57644220e1ac89afbc8b145a8c57791"}
Sep 17 19:13:29.079724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:29.079696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5" event={"ID":"a33db913-5dc8-4533-a6de-7b071065a909","Type":"ContainerStarted","Data":"2b9c4107db5a99eb1269b82f8f8207da4b5b5ed1e80f7daa254b27d1304e6452"}
Sep 17 19:13:30.086275 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.086242    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerID="1e4d86eb39bbc236979eb697f8184a66916db8793689a02e208b91aaab36e4c0" exitCode=0
Sep 17 19:13:30.086661 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.086332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg" event={"ID":"2ffac514-fcbd-4177-bc22-ca4f585f55bf","Type":"ContainerDied","Data":"1e4d86eb39bbc236979eb697f8184a66916db8793689a02e208b91aaab36e4c0"}
Sep 17 19:13:30.088464 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.088439    2583 generic.go:356] "Generic (PLEG): container finished" podID="a33db913-5dc8-4533-a6de-7b071065a909" containerID="2b9c4107db5a99eb1269b82f8f8207da4b5b5ed1e80f7daa254b27d1304e6452" exitCode=0
Sep 17 19:13:30.088569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.088527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5" event={"ID":"a33db913-5dc8-4533-a6de-7b071065a909","Type":"ContainerDied","Data":"2b9c4107db5a99eb1269b82f8f8207da4b5b5ed1e80f7daa254b27d1304e6452"}
Sep 17 19:13:30.090799 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.090775    2583 generic.go:356] "Generic (PLEG): container finished" podID="b6a24697-6a01-4808-acee-132cff3f8208" containerID="85d6713173c32f373efa5772950f8f9de679dc0fad0de8f052abaaf5c56ad501" exitCode=0
Sep 17 19:13:30.090913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.090888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69" event={"ID":"b6a24697-6a01-4808-acee-132cff3f8208","Type":"ContainerDied","Data":"85d6713173c32f373efa5772950f8f9de679dc0fad0de8f052abaaf5c56ad501"}
Sep 17 19:13:30.092981 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.092958    2583 generic.go:356] "Generic (PLEG): container finished" podID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerID="ddb6793e2884f3ebb16ea163851be8d1a57644220e1ac89afbc8b145a8c57791" exitCode=0
Sep 17 19:13:30.093078 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:30.093005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq" event={"ID":"f9888543-821e-4c89-a6cd-216c34f1bd8a","Type":"ContainerDied","Data":"ddb6793e2884f3ebb16ea163851be8d1a57644220e1ac89afbc8b145a8c57791"}
Sep 17 19:13:31.100172 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.100131    2583 generic.go:356] "Generic (PLEG): container finished" podID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerID="a496bd78a41fd1fbdfd4fcac83a80badc2f530ce18508ed92bbc4ca2caf693a3" exitCode=0
Sep 17 19:13:31.100591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.100230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq" event={"ID":"f9888543-821e-4c89-a6cd-216c34f1bd8a","Type":"ContainerDied","Data":"a496bd78a41fd1fbdfd4fcac83a80badc2f530ce18508ed92bbc4ca2caf693a3"}
Sep 17 19:13:31.102524 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.102498    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerID="0ed0dc08c9be25898dc5e46c68cb80ad0e6f96b0708682460858be384383c3ff" exitCode=0
Sep 17 19:13:31.102625 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.102585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg" event={"ID":"2ffac514-fcbd-4177-bc22-ca4f585f55bf","Type":"ContainerDied","Data":"0ed0dc08c9be25898dc5e46c68cb80ad0e6f96b0708682460858be384383c3ff"}
Sep 17 19:13:31.104980 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.104952    2583 generic.go:356] "Generic (PLEG): container finished" podID="a33db913-5dc8-4533-a6de-7b071065a909" containerID="1bf5bb81f3fe20fb7606798dc8ac3b686ebb7ded3275991a89bd60381102ef09" exitCode=0
Sep 17 19:13:31.105069 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.104994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5" event={"ID":"a33db913-5dc8-4533-a6de-7b071065a909","Type":"ContainerDied","Data":"1bf5bb81f3fe20fb7606798dc8ac3b686ebb7ded3275991a89bd60381102ef09"}
Sep 17 19:13:31.107231 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.107210    2583 generic.go:356] "Generic (PLEG): container finished" podID="b6a24697-6a01-4808-acee-132cff3f8208" containerID="b888e29713a29f6a7e0c7478fc1f64d960a63dac277d96be9f9e83e00979a37e" exitCode=0
Sep 17 19:13:31.107303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:31.107252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69" event={"ID":"b6a24697-6a01-4808-acee-132cff3f8208","Type":"ContainerDied","Data":"b888e29713a29f6a7e0c7478fc1f64d960a63dac277d96be9f9e83e00979a37e"}
Sep 17 19:13:32.248752 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.248726    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:32.282727 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.282702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-bundle\") pod \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") "
Sep 17 19:13:32.282856 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.282757    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpjbp\" (UniqueName: \"kubernetes.io/projected/2ffac514-fcbd-4177-bc22-ca4f585f55bf-kube-api-access-lpjbp\") pod \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") "
Sep 17 19:13:32.282925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.282863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-util\") pod \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\" (UID: \"2ffac514-fcbd-4177-bc22-ca4f585f55bf\") "
Sep 17 19:13:32.283540 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.283505    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-bundle" (OuterVolumeSpecName: "bundle") pod "2ffac514-fcbd-4177-bc22-ca4f585f55bf" (UID: "2ffac514-fcbd-4177-bc22-ca4f585f55bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.286877 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.286849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffac514-fcbd-4177-bc22-ca4f585f55bf-kube-api-access-lpjbp" (OuterVolumeSpecName: "kube-api-access-lpjbp") pod "2ffac514-fcbd-4177-bc22-ca4f585f55bf" (UID: "2ffac514-fcbd-4177-bc22-ca4f585f55bf"). InnerVolumeSpecName "kube-api-access-lpjbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:32.290010 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.289988    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-util" (OuterVolumeSpecName: "util") pod "2ffac514-fcbd-4177-bc22-ca4f585f55bf" (UID: "2ffac514-fcbd-4177-bc22-ca4f585f55bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.292400 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.292376    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:32.303756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.303734    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:32.306756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.306737    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:32.383482 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383410    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lw4j\" (UniqueName: \"kubernetes.io/projected/b6a24697-6a01-4808-acee-132cff3f8208-kube-api-access-2lw4j\") pod \"b6a24697-6a01-4808-acee-132cff3f8208\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") "
Sep 17 19:13:32.383482 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-bundle\") pod \"b6a24697-6a01-4808-acee-132cff3f8208\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") "
Sep 17 19:13:32.383681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-bundle\") pod \"f9888543-821e-4c89-a6cd-216c34f1bd8a\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") "
Sep 17 19:13:32.383681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383517    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-util\") pod \"b6a24697-6a01-4808-acee-132cff3f8208\" (UID: \"b6a24697-6a01-4808-acee-132cff3f8208\") "
Sep 17 19:13:32.383681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383564    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zs2k\" (UniqueName: \"kubernetes.io/projected/f9888543-821e-4c89-a6cd-216c34f1bd8a-kube-api-access-5zs2k\") pod \"f9888543-821e-4c89-a6cd-216c34f1bd8a\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") "
Sep 17 19:13:32.383681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383592    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-util\") pod \"a33db913-5dc8-4533-a6de-7b071065a909\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") "
Sep 17 19:13:32.383681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-util\") pod \"f9888543-821e-4c89-a6cd-216c34f1bd8a\" (UID: \"f9888543-821e-4c89-a6cd-216c34f1bd8a\") "
Sep 17 19:13:32.383955 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-bundle\") pod \"a33db913-5dc8-4533-a6de-7b071065a909\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") "
Sep 17 19:13:32.383955 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74hfx\" (UniqueName: \"kubernetes.io/projected/a33db913-5dc8-4533-a6de-7b071065a909-kube-api-access-74hfx\") pod \"a33db913-5dc8-4533-a6de-7b071065a909\" (UID: \"a33db913-5dc8-4533-a6de-7b071065a909\") "
Sep 17 19:13:32.384056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383968    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.384056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.383986    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjbp\" (UniqueName: \"kubernetes.io/projected/2ffac514-fcbd-4177-bc22-ca4f585f55bf-kube-api-access-lpjbp\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.384056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.384002    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffac514-fcbd-4177-bc22-ca4f585f55bf-util\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.384220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.384120    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-bundle" (OuterVolumeSpecName: "bundle") pod "f9888543-821e-4c89-a6cd-216c34f1bd8a" (UID: "f9888543-821e-4c89-a6cd-216c34f1bd8a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.384866 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.384824    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-bundle" (OuterVolumeSpecName: "bundle") pod "a33db913-5dc8-4533-a6de-7b071065a909" (UID: "a33db913-5dc8-4533-a6de-7b071065a909"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.385385 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.385363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-bundle" (OuterVolumeSpecName: "bundle") pod "b6a24697-6a01-4808-acee-132cff3f8208" (UID: "b6a24697-6a01-4808-acee-132cff3f8208"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.386080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.386052    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9888543-821e-4c89-a6cd-216c34f1bd8a-kube-api-access-5zs2k" (OuterVolumeSpecName: "kube-api-access-5zs2k") pod "f9888543-821e-4c89-a6cd-216c34f1bd8a" (UID: "f9888543-821e-4c89-a6cd-216c34f1bd8a"). InnerVolumeSpecName "kube-api-access-5zs2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:32.386187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.386155    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a24697-6a01-4808-acee-132cff3f8208-kube-api-access-2lw4j" (OuterVolumeSpecName: "kube-api-access-2lw4j") pod "b6a24697-6a01-4808-acee-132cff3f8208" (UID: "b6a24697-6a01-4808-acee-132cff3f8208"). InnerVolumeSpecName "kube-api-access-2lw4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:32.386187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.386164    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33db913-5dc8-4533-a6de-7b071065a909-kube-api-access-74hfx" (OuterVolumeSpecName: "kube-api-access-74hfx") pod "a33db913-5dc8-4533-a6de-7b071065a909" (UID: "a33db913-5dc8-4533-a6de-7b071065a909"). InnerVolumeSpecName "kube-api-access-74hfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:32.390546 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.390507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-util" (OuterVolumeSpecName: "util") pod "a33db913-5dc8-4533-a6de-7b071065a909" (UID: "a33db913-5dc8-4533-a6de-7b071065a909"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.392890 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.392863    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-util" (OuterVolumeSpecName: "util") pod "f9888543-821e-4c89-a6cd-216c34f1bd8a" (UID: "f9888543-821e-4c89-a6cd-216c34f1bd8a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.392954 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.392938    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-util" (OuterVolumeSpecName: "util") pod "b6a24697-6a01-4808-acee-132cff3f8208" (UID: "b6a24697-6a01-4808-acee-132cff3f8208"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:32.485266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485244    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-util\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485266    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zs2k\" (UniqueName: \"kubernetes.io/projected/f9888543-821e-4c89-a6cd-216c34f1bd8a-kube-api-access-5zs2k\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485276    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-util\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485284    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-util\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485292    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a33db913-5dc8-4533-a6de-7b071065a909-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485300    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74hfx\" (UniqueName: \"kubernetes.io/projected/a33db913-5dc8-4533-a6de-7b071065a909-kube-api-access-74hfx\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485307    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lw4j\" (UniqueName: \"kubernetes.io/projected/b6a24697-6a01-4808-acee-132cff3f8208-kube-api-access-2lw4j\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485315    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a24697-6a01-4808-acee-132cff3f8208-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:32.485393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:32.485323    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9888543-821e-4c89-a6cd-216c34f1bd8a-bundle\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:13:33.122333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.122308    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69"
Sep 17 19:13:33.122494 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.122342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cn6s69" event={"ID":"b6a24697-6a01-4808-acee-132cff3f8208","Type":"ContainerDied","Data":"56727226cb6db70ccc50edb255903c81088ffe952b7de1cfdec4f61c09aca2a0"}
Sep 17 19:13:33.122494 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.122404    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56727226cb6db70ccc50edb255903c81088ffe952b7de1cfdec4f61c09aca2a0"
Sep 17 19:13:33.124559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.124536    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq"
Sep 17 19:13:33.124676 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.124569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7q29lq" event={"ID":"f9888543-821e-4c89-a6cd-216c34f1bd8a","Type":"ContainerDied","Data":"57f0acd6485c5d6544bbd243976515867452d589c56d904c6e036ebaffb726ab"}
Sep 17 19:13:33.124676 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.124616    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57f0acd6485c5d6544bbd243976515867452d589c56d904c6e036ebaffb726ab"
Sep 17 19:13:33.126889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.126867    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg"
Sep 17 19:13:33.127000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.126892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axtxkg" event={"ID":"2ffac514-fcbd-4177-bc22-ca4f585f55bf","Type":"ContainerDied","Data":"e175a9eb38a86a15ed3feedae65c9780bac89af1d12d44b3db046ae7f153127f"}
Sep 17 19:13:33.127000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.126933    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e175a9eb38a86a15ed3feedae65c9780bac89af1d12d44b3db046ae7f153127f"
Sep 17 19:13:33.129083 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.129064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5" event={"ID":"a33db913-5dc8-4533-a6de-7b071065a909","Type":"ContainerDied","Data":"538ed0f00a085c726c51f9a1772c7f20562d79221304af233ae1ee1f2b5326fb"}
Sep 17 19:13:33.129206 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.129093    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538ed0f00a085c726c51f9a1772c7f20562d79221304af233ae1ee1f2b5326fb"
Sep 17 19:13:33.129206 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:33.129133    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hcjd5"
Sep 17 19:13:37.553215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553185    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"]
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553508    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerName="pull"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553519    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerName="pull"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553531    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33db913-5dc8-4533-a6de-7b071065a909" containerName="extract"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553536    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33db913-5dc8-4533-a6de-7b071065a909" containerName="extract"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553542    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33db913-5dc8-4533-a6de-7b071065a909" containerName="pull"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553548    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33db913-5dc8-4533-a6de-7b071065a909" containerName="pull"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553555    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a24697-6a01-4808-acee-132cff3f8208" containerName="pull"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553560    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a24697-6a01-4808-acee-132cff3f8208" containerName="pull"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553566    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33db913-5dc8-4533-a6de-7b071065a909" containerName="util"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553571    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33db913-5dc8-4533-a6de-7b071065a909" containerName="util"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553576    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerName="extract"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553580    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerName="extract"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553589    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerName="util"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553593    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerName="util"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553598    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerName="pull"
Sep 17 19:13:37.553600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553605    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerName="pull"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553611    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerName="extract"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553616    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerName="extract"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553623    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a24697-6a01-4808-acee-132cff3f8208" containerName="util"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553628    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a24697-6a01-4808-acee-132cff3f8208" containerName="util"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553635    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerName="util"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553640    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerName="util"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553645    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a24697-6a01-4808-acee-132cff3f8208" containerName="extract"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553650    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a24697-6a01-4808-acee-132cff3f8208" containerName="extract"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553719    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33db913-5dc8-4533-a6de-7b071065a909" containerName="extract"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553734    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffac514-fcbd-4177-bc22-ca4f585f55bf" containerName="extract"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553741    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a24697-6a01-4808-acee-132cff3f8208" containerName="extract"
Sep 17 19:13:37.554089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.553748    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9888543-821e-4c89-a6cd-216c34f1bd8a" containerName="extract"
Sep 17 19:13:37.556279 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.556262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"
Sep 17 19:13:37.559057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.559038    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 19:13:37.560059 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.560039    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 19:13:37.560140 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.560066    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-486hn\""
Sep 17 19:13:37.566870 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.566837    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"]
Sep 17 19:13:37.618836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.618813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqr57\" (UniqueName: \"kubernetes.io/projected/16ae978c-1626-473a-8b4b-aa0b04747f98-kube-api-access-gqr57\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-hx9zc\" (UID: \"16ae978c-1626-473a-8b4b-aa0b04747f98\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"
Sep 17 19:13:37.720001 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.719974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqr57\" (UniqueName: \"kubernetes.io/projected/16ae978c-1626-473a-8b4b-aa0b04747f98-kube-api-access-gqr57\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-hx9zc\" (UID: \"16ae978c-1626-473a-8b4b-aa0b04747f98\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"
Sep 17 19:13:37.734367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.734337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqr57\" (UniqueName: \"kubernetes.io/projected/16ae978c-1626-473a-8b4b-aa0b04747f98-kube-api-access-gqr57\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-hx9zc\" (UID: \"16ae978c-1626-473a-8b4b-aa0b04747f98\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"
Sep 17 19:13:37.867002 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:37.866947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"
Sep 17 19:13:38.156314 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:38.156290    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"]
Sep 17 19:13:38.158400 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:38.158378    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ae978c_1626_473a_8b4b_aa0b04747f98.slice/crio-5d2286147e0be2167d7171a0e560dc61c92c6e211a81fc2dfbd3a27194619f86 WatchSource:0}: Error finding container 5d2286147e0be2167d7171a0e560dc61c92c6e211a81fc2dfbd3a27194619f86: Status 404 returned error can't find the container with id 5d2286147e0be2167d7171a0e560dc61c92c6e211a81fc2dfbd3a27194619f86
Sep 17 19:13:39.162098 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:39.162045    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc" event={"ID":"16ae978c-1626-473a-8b4b-aa0b04747f98","Type":"ContainerStarted","Data":"5d2286147e0be2167d7171a0e560dc61c92c6e211a81fc2dfbd3a27194619f86"}
Sep 17 19:13:40.171660 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:40.171603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc" event={"ID":"16ae978c-1626-473a-8b4b-aa0b04747f98","Type":"ContainerStarted","Data":"9f97e574ff851bafda140a8f171edccf3caeb791011de1df42a665cb720f7c28"}
Sep 17 19:13:40.172049 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:40.171719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"
Sep 17 19:13:41.756371 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.756322    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc" podStartSLOduration=3.072803596 podStartE2EDuration="4.756302427s" totalImagesPullingTime="1.683498831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:40.21322156 +0000 UTC m=+422.261101429" watchObservedRunningTime="2026-09-17 19:13:41.756302427 +0000 UTC m=+423.804182287"
Sep 17 19:13:41.757201 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.757180    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"]
Sep 17 19:13:41.761996 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.761978    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:41.765417 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.765398    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5kjx6\""
Sep 17 19:13:41.774912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.774889    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"]
Sep 17 19:13:41.865519 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.865499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqxv\" (UniqueName: \"kubernetes.io/projected/24fe1c6f-7a85-4a70-b4aa-b76bab1314d2-kube-api-access-5bqxv\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xd5cs\" (UID: \"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:41.865611 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.865560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24fe1c6f-7a85-4a70-b4aa-b76bab1314d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xd5cs\" (UID: \"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:41.966517 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.966493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24fe1c6f-7a85-4a70-b4aa-b76bab1314d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xd5cs\" (UID: \"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:41.966610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.966548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqxv\" (UniqueName: \"kubernetes.io/projected/24fe1c6f-7a85-4a70-b4aa-b76bab1314d2-kube-api-access-5bqxv\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xd5cs\" (UID: \"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:41.966946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.966929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24fe1c6f-7a85-4a70-b4aa-b76bab1314d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xd5cs\" (UID: \"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:41.976938 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:41.976913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqxv\" (UniqueName: \"kubernetes.io/projected/24fe1c6f-7a85-4a70-b4aa-b76bab1314d2-kube-api-access-5bqxv\") pod \"kuadrant-operator-controller-manager-6d4cb79c-xd5cs\" (UID: \"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:42.072979 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:42.072953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:42.370139 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:42.370115    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"]
Sep 17 19:13:42.372169 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:42.372134    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fe1c6f_7a85_4a70_b4aa_b76bab1314d2.slice/crio-11e16c5d764adedf54638ed719cb5da9ed41f3f8c5976240ebe0beac9d0a4593 WatchSource:0}: Error finding container 11e16c5d764adedf54638ed719cb5da9ed41f3f8c5976240ebe0beac9d0a4593: Status 404 returned error can't find the container with id 11e16c5d764adedf54638ed719cb5da9ed41f3f8c5976240ebe0beac9d0a4593
Sep 17 19:13:43.189061 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:43.189025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs" event={"ID":"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2","Type":"ContainerStarted","Data":"11e16c5d764adedf54638ed719cb5da9ed41f3f8c5976240ebe0beac9d0a4593"}
Sep 17 19:13:47.020382 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.020349    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"]
Sep 17 19:13:47.023237 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.023214    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:13:47.027449 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:13:47.027419    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"dns-operator-controller-manager-65dc8f76f5-8pjhs\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="d464d6ee-96bd-4e67-9476-854efbba0abd" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:13:47.027980 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:13:47.027947    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-operator-controller-env\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" type="*v1.ConfigMap"
Sep 17 19:13:47.028081 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:13:47.028050    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-operator-controller-manager-dockercfg-lcbl7\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lcbl7\"" type="*v1.Secret"
Sep 17 19:13:47.052133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.052112    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"]
Sep 17 19:13:47.108742 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.108709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46g6\" (UniqueName: \"kubernetes.io/projected/d464d6ee-96bd-4e67-9476-854efbba0abd-kube-api-access-k46g6\") pod \"dns-operator-controller-manager-65dc8f76f5-8pjhs\" (UID: \"d464d6ee-96bd-4e67-9476-854efbba0abd\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:13:47.209990 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.209961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46g6\" (UniqueName: \"kubernetes.io/projected/d464d6ee-96bd-4e67-9476-854efbba0abd-kube-api-access-k46g6\") pod \"dns-operator-controller-manager-65dc8f76f5-8pjhs\" (UID: \"d464d6ee-96bd-4e67-9476-854efbba0abd\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:13:47.216309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.216279    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs" event={"ID":"24fe1c6f-7a85-4a70-b4aa-b76bab1314d2","Type":"ContainerStarted","Data":"5ba0e66eecc8ec833114abe7594539bf469fde7d6c906ace2426d95f17a27c33"}
Sep 17 19:13:47.216436 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.216422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:13:47.224140 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.224112    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46g6\" (UniqueName: \"kubernetes.io/projected/d464d6ee-96bd-4e67-9476-854efbba0abd-kube-api-access-k46g6\") pod \"dns-operator-controller-manager-65dc8f76f5-8pjhs\" (UID: \"d464d6ee-96bd-4e67-9476-854efbba0abd\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:13:47.266016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.265979    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs" podStartSLOduration=2.409305811 podStartE2EDuration="6.265965015s" totalImagesPullingTime="3.856659204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:47.263791652 +0000 UTC m=+429.311671509" watchObservedRunningTime="2026-09-17 19:13:47.265965015 +0000 UTC m=+429.313844926"
Sep 17 19:13:47.916360 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:47.916330    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 19:13:48.320497 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.320473    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lcbl7\""
Sep 17 19:13:48.325825 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.325809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:13:48.606338 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.606294    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"]
Sep 17 19:13:48.607810 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:48.607780    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd464d6ee_96bd_4e67_9476_854efbba0abd.slice/crio-5066d8cc7c0d1d713277b2f5e997fb88a6def1eb262497bcd93722ef6236eb12 WatchSource:0}: Error finding container 5066d8cc7c0d1d713277b2f5e997fb88a6def1eb262497bcd93722ef6236eb12: Status 404 returned error can't find the container with id 5066d8cc7c0d1d713277b2f5e997fb88a6def1eb262497bcd93722ef6236eb12
Sep 17 19:13:48.609331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.609303    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"]
Sep 17 19:13:48.612179 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.612163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.614753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.614734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 19:13:48.614916 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.614899    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 19:13:48.614987 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.614973    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gfjzn\""
Sep 17 19:13:48.623711 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.623676    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"]
Sep 17 19:13:48.720795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.720768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/859ac01d-89f9-4947-b62c-7806c5833e59-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.720886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.720808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/859ac01d-89f9-4947-b62c-7806c5833e59-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.720944 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.720885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpb4\" (UniqueName: \"kubernetes.io/projected/859ac01d-89f9-4947-b62c-7806c5833e59-kube-api-access-vjpb4\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.821309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.821250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpb4\" (UniqueName: \"kubernetes.io/projected/859ac01d-89f9-4947-b62c-7806c5833e59-kube-api-access-vjpb4\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.821309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.821292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/859ac01d-89f9-4947-b62c-7806c5833e59-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.821468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.821325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/859ac01d-89f9-4947-b62c-7806c5833e59-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.821962 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.821944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/859ac01d-89f9-4947-b62c-7806c5833e59-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.823915 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.823897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/859ac01d-89f9-4947-b62c-7806c5833e59-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.832492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.832463    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpb4\" (UniqueName: \"kubernetes.io/projected/859ac01d-89f9-4947-b62c-7806c5833e59-kube-api-access-vjpb4\") pod \"kuadrant-console-plugin-fbdd4f445-9qrk2\" (UID: \"859ac01d-89f9-4947-b62c-7806c5833e59\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:48.922388 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:48.922333    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"
Sep 17 19:13:49.191061 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:49.191037    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2"]
Sep 17 19:13:49.192752 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:13:49.192728    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod859ac01d_89f9_4947_b62c_7806c5833e59.slice/crio-79e3ff9fc3f956db295ae623bcfe364c5dae73b56371340e846283576184e535 WatchSource:0}: Error finding container 79e3ff9fc3f956db295ae623bcfe364c5dae73b56371340e846283576184e535: Status 404 returned error can't find the container with id 79e3ff9fc3f956db295ae623bcfe364c5dae73b56371340e846283576184e535
Sep 17 19:13:49.227271 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:49.227245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs" event={"ID":"d464d6ee-96bd-4e67-9476-854efbba0abd","Type":"ContainerStarted","Data":"5066d8cc7c0d1d713277b2f5e997fb88a6def1eb262497bcd93722ef6236eb12"}
Sep 17 19:13:49.228458 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:49.228437    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2" event={"ID":"859ac01d-89f9-4947-b62c-7806c5833e59","Type":"ContainerStarted","Data":"79e3ff9fc3f956db295ae623bcfe364c5dae73b56371340e846283576184e535"}
Sep 17 19:13:51.179539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:51.179508    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-hx9zc"
Sep 17 19:13:51.244777 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:51.244735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs" event={"ID":"d464d6ee-96bd-4e67-9476-854efbba0abd","Type":"ContainerStarted","Data":"d1877548a43eeaa6b71a35418d541bc58216cb8d2afd08f5c031749fc1a6f199"}
Sep 17 19:13:51.244947 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:51.244804    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:13:51.267184 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:51.267128    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs" podStartSLOduration=3.236938867 podStartE2EDuration="5.267110147s" totalImagesPullingTime="2.03017128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:51.264048439 +0000 UTC m=+433.311928295" watchObservedRunningTime="2026-09-17 19:13:51.267110147 +0000 UTC m=+433.314990006"
Sep 17 19:13:54.266643 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:54.266566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2" event={"ID":"859ac01d-89f9-4947-b62c-7806c5833e59","Type":"ContainerStarted","Data":"d5238ab0876959083b8e116558d55252524ee71909088fff505ea371c0f36253"}
Sep 17 19:13:54.288969 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:54.288927    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9qrk2" podStartSLOduration=1.616731239 podStartE2EDuration="6.288913569s" totalImagesPullingTime="4.67218233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:54.287208159 +0000 UTC m=+436.335088017" watchObservedRunningTime="2026-09-17 19:13:54.288913569 +0000 UTC m=+436.336793434"
Sep 17 19:13:58.224764 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:13:58.224726    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-xd5cs"
Sep 17 19:14:02.256301 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:02.256265    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-8pjhs"
Sep 17 19:14:30.836460 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:30.836423    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"]
Sep 17 19:14:30.853763 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:30.853732    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"]
Sep 17 19:14:30.853925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:30.853825    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:30.856710 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:30.856660    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 19:14:30.874102 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:30.874070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"]
Sep 17 19:14:30.957554 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:30.957528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdjh\" (UniqueName: \"kubernetes.io/projected/0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773-kube-api-access-wkdjh\") pod \"limitador-limitador-84994f7ffc-nlwm5\" (UID: \"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:30.957651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:30.957581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773-config-file\") pod \"limitador-limitador-84994f7ffc-nlwm5\" (UID: \"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:31.058955 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:31.058927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdjh\" (UniqueName: \"kubernetes.io/projected/0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773-kube-api-access-wkdjh\") pod \"limitador-limitador-84994f7ffc-nlwm5\" (UID: \"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:31.059046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:31.058988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773-config-file\") pod \"limitador-limitador-84994f7ffc-nlwm5\" (UID: \"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:31.059610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:31.059593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773-config-file\") pod \"limitador-limitador-84994f7ffc-nlwm5\" (UID: \"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:31.068491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:31.068469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdjh\" (UniqueName: \"kubernetes.io/projected/0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773-kube-api-access-wkdjh\") pod \"limitador-limitador-84994f7ffc-nlwm5\" (UID: \"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:31.164438 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:31.164388    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:31.436886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:31.436859    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"]
Sep 17 19:14:31.438468 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:14:31.438444    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d377d7b_5d46_4d39_a2a6_5cc0f1aaf773.slice/crio-dfe51628aad389932254e35c2bf7c29a0224d2d2cb555593bdb5ca701a5ce9f2 WatchSource:0}: Error finding container dfe51628aad389932254e35c2bf7c29a0224d2d2cb555593bdb5ca701a5ce9f2: Status 404 returned error can't find the container with id dfe51628aad389932254e35c2bf7c29a0224d2d2cb555593bdb5ca701a5ce9f2
Sep 17 19:14:31.460921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:31.460887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5" event={"ID":"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773","Type":"ContainerStarted","Data":"dfe51628aad389932254e35c2bf7c29a0224d2d2cb555593bdb5ca701a5ce9f2"}
Sep 17 19:14:33.473369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:33.473327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5" event={"ID":"0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773","Type":"ContainerStarted","Data":"cc7131615e58d280119d4577de185b687341c5fe3056d1108eb8d823ad9f0111"}
Sep 17 19:14:33.473906 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:33.473478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:14:33.498797 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:33.498750    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5" podStartSLOduration=2.105601607 podStartE2EDuration="3.498734893s" totalImagesPullingTime="1.393133286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:33.497743231 +0000 UTC m=+475.545623123" watchObservedRunningTime="2026-09-17 19:14:33.498734893 +0000 UTC m=+475.546614749"
Sep 17 19:14:44.479424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:14:44.479389    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-nlwm5"
Sep 17 19:15:17.327961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.327920    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"]
Sep 17 19:15:17.328490 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.328128    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7" podUID="d3168d53-2398-4744-99ae-4f5b21c10024" containerName="discovery" containerID="cri-o://71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61" gracePeriod=30
Sep 17 19:15:17.572903 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.572880    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:15:17.710125 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.710050    2583 generic.go:356] "Generic (PLEG): container finished" podID="d3168d53-2398-4744-99ae-4f5b21c10024" containerID="71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61" exitCode=0
Sep 17 19:15:17.710125 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.710108    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"
Sep 17 19:15:17.710268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.710125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7" event={"ID":"d3168d53-2398-4744-99ae-4f5b21c10024","Type":"ContainerDied","Data":"71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61"}
Sep 17 19:15:17.710268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.710173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7" event={"ID":"d3168d53-2398-4744-99ae-4f5b21c10024","Type":"ContainerDied","Data":"2fd4f3a81505ec62a5e2f08ec8f92392164594fd5eb349710447a84d1dd25861"}
Sep 17 19:15:17.710268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.710192    2583 scope.go:117] "RemoveContainer" containerID="71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61"
Sep 17 19:15:17.716449 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716425    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb4rq\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-kube-api-access-hb4rq\") pod \"d3168d53-2398-4744-99ae-4f5b21c10024\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") "
Sep 17 19:15:17.716533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-cacerts\") pod \"d3168d53-2398-4744-99ae-4f5b21c10024\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") "
Sep 17 19:15:17.716533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716481    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-dns-cert\") pod \"d3168d53-2398-4744-99ae-4f5b21c10024\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") "
Sep 17 19:15:17.716533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716512    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3168d53-2398-4744-99ae-4f5b21c10024-local-certs\") pod \"d3168d53-2398-4744-99ae-4f5b21c10024\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") "
Sep 17 19:15:17.716698 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716536    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-ca-configmap\") pod \"d3168d53-2398-4744-99ae-4f5b21c10024\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") "
Sep 17 19:15:17.716698 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716581    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-kubeconfig\") pod \"d3168d53-2398-4744-99ae-4f5b21c10024\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") "
Sep 17 19:15:17.716698 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-istio-token\") pod \"d3168d53-2398-4744-99ae-4f5b21c10024\" (UID: \"d3168d53-2398-4744-99ae-4f5b21c10024\") "
Sep 17 19:15:17.717068 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.716989    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d3168d53-2398-4744-99ae-4f5b21c10024" (UID: "d3168d53-2398-4744-99ae-4f5b21c10024"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:15:17.721242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.721177    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d3168d53-2398-4744-99ae-4f5b21c10024" (UID: "d3168d53-2398-4744-99ae-4f5b21c10024"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:15:17.721242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.721189    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3168d53-2398-4744-99ae-4f5b21c10024" (UID: "d3168d53-2398-4744-99ae-4f5b21c10024"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:15:17.721242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.721184    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-kube-api-access-hb4rq" (OuterVolumeSpecName: "kube-api-access-hb4rq") pod "d3168d53-2398-4744-99ae-4f5b21c10024" (UID: "d3168d53-2398-4744-99ae-4f5b21c10024"). InnerVolumeSpecName "kube-api-access-hb4rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:15:17.721242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.721198    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d3168d53-2398-4744-99ae-4f5b21c10024" (UID: "d3168d53-2398-4744-99ae-4f5b21c10024"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:15:17.721492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.721247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3168d53-2398-4744-99ae-4f5b21c10024-local-certs" (OuterVolumeSpecName: "local-certs") pod "d3168d53-2398-4744-99ae-4f5b21c10024" (UID: "d3168d53-2398-4744-99ae-4f5b21c10024"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:17.721550 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.721525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-cacerts" (OuterVolumeSpecName: "cacerts") pod "d3168d53-2398-4744-99ae-4f5b21c10024" (UID: "d3168d53-2398-4744-99ae-4f5b21c10024"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:15:17.721974 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.721895    2583 scope.go:117] "RemoveContainer" containerID="71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61"
Sep 17 19:15:17.723234 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:15:17.722814    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61\": container with ID starting with 71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61 not found: ID does not exist" containerID="71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61"
Sep 17 19:15:17.723234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.722849    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61"} err="failed to get container status \"71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61\": rpc error: code = NotFound desc = could not find container \"71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61\": container with ID starting with 71dce9ac993541965d9c4823835be3a9a826f479996ca0306782a82b11219b61 not found: ID does not exist"
Sep 17 19:15:17.818046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.818025    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-ca-configmap\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:15:17.818046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.818044    2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-kubeconfig\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:15:17.818174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.818053    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-istio-token\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:15:17.818174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.818061    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4rq\" (UniqueName: \"kubernetes.io/projected/d3168d53-2398-4744-99ae-4f5b21c10024-kube-api-access-hb4rq\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:15:17.818174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.818069    2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-cacerts\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:15:17.818174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.818077    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3168d53-2398-4744-99ae-4f5b21c10024-istio-csr-dns-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:15:17.818174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:17.818086    2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3168d53-2398-4744-99ae-4f5b21c10024-local-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:15:18.033138 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:18.033101    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"]
Sep 17 19:15:18.038390 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:18.038367    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lq7q7"]
Sep 17 19:15:18.256146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:18.256116    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3168d53-2398-4744-99ae-4f5b21c10024" path="/var/lib/kubelet/pods/d3168d53-2398-4744-99ae-4f5b21c10024/volumes"
Sep 17 19:15:25.400157 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.400080    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-5gkql"]
Sep 17 19:15:25.400574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.400431    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3168d53-2398-4744-99ae-4f5b21c10024" containerName="discovery"
Sep 17 19:15:25.400574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.400443    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3168d53-2398-4744-99ae-4f5b21c10024" containerName="discovery"
Sep 17 19:15:25.400574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.400503    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3168d53-2398-4744-99ae-4f5b21c10024" containerName="discovery"
Sep 17 19:15:25.403724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.403676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.407617 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.407594    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 19:15:25.407617 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.407614    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-dlrfd\""
Sep 17 19:15:25.407804 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.407596    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 19:15:25.407804 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.407594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 19:15:25.415265 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.415242    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-5gkql"]
Sep 17 19:15:25.429812 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.429792    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"]
Sep 17 19:15:25.433060 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.433036    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:25.435591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.435574    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 19:15:25.435667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.435576    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-qcvdl\""
Sep 17 19:15:25.442986 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.442964    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"]
Sep 17 19:15:25.453624 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.453602    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-sr7zz"]
Sep 17 19:15:25.457002 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.456984    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.459451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.459433    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 19:15:25.459554 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.459436    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5r95b\""
Sep 17 19:15:25.466711 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.466630    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sr7zz"]
Sep 17 19:15:25.576538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.576517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f7974ad-93f1-4883-9f1a-43c708c6b4cb-data\") pod \"seaweedfs-64568bcd49-sr7zz\" (UID: \"4f7974ad-93f1-4883-9f1a-43c708c6b4cb\") " pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.576652 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.576553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094fe571-f8e5-4841-8b88-feca944b2015-cert\") pod \"llmisvc-controller-manager-b64f7dcc6-7mn8m\" (UID: \"094fe571-f8e5-4841-8b88-feca944b2015\") " pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:25.576652 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.576575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q489\" (UniqueName: \"kubernetes.io/projected/094fe571-f8e5-4841-8b88-feca944b2015-kube-api-access-7q489\") pod \"llmisvc-controller-manager-b64f7dcc6-7mn8m\" (UID: \"094fe571-f8e5-4841-8b88-feca944b2015\") " pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:25.576756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.576664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-cert\") pod \"kserve-controller-manager-66576f5db6-5gkql\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") " pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.576756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.576746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pz6s\" (UniqueName: \"kubernetes.io/projected/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-kube-api-access-8pz6s\") pod \"kserve-controller-manager-66576f5db6-5gkql\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") " pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.576820 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.576767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjvk\" (UniqueName: \"kubernetes.io/projected/4f7974ad-93f1-4883-9f1a-43c708c6b4cb-kube-api-access-2sjvk\") pod \"seaweedfs-64568bcd49-sr7zz\" (UID: \"4f7974ad-93f1-4883-9f1a-43c708c6b4cb\") " pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.677964 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.677896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pz6s\" (UniqueName: \"kubernetes.io/projected/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-kube-api-access-8pz6s\") pod \"kserve-controller-manager-66576f5db6-5gkql\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") " pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.677964 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.677926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjvk\" (UniqueName: \"kubernetes.io/projected/4f7974ad-93f1-4883-9f1a-43c708c6b4cb-kube-api-access-2sjvk\") pod \"seaweedfs-64568bcd49-sr7zz\" (UID: \"4f7974ad-93f1-4883-9f1a-43c708c6b4cb\") " pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.677964 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.677955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f7974ad-93f1-4883-9f1a-43c708c6b4cb-data\") pod \"seaweedfs-64568bcd49-sr7zz\" (UID: \"4f7974ad-93f1-4883-9f1a-43c708c6b4cb\") " pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.678131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.678066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094fe571-f8e5-4841-8b88-feca944b2015-cert\") pod \"llmisvc-controller-manager-b64f7dcc6-7mn8m\" (UID: \"094fe571-f8e5-4841-8b88-feca944b2015\") " pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:25.678131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.678104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q489\" (UniqueName: \"kubernetes.io/projected/094fe571-f8e5-4841-8b88-feca944b2015-kube-api-access-7q489\") pod \"llmisvc-controller-manager-b64f7dcc6-7mn8m\" (UID: \"094fe571-f8e5-4841-8b88-feca944b2015\") " pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:25.678227 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.678150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-cert\") pod \"kserve-controller-manager-66576f5db6-5gkql\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") " pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.678227 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:15:25.678204    2583 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 19:15:25.678327 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:15:25.678263    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/094fe571-f8e5-4841-8b88-feca944b2015-cert podName:094fe571-f8e5-4841-8b88-feca944b2015 nodeName:}" failed. No retries permitted until 2026-09-17 19:15:26.178243464 +0000 UTC m=+528.226123300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/094fe571-f8e5-4841-8b88-feca944b2015-cert") pod "llmisvc-controller-manager-b64f7dcc6-7mn8m" (UID: "094fe571-f8e5-4841-8b88-feca944b2015") : secret "llmisvc-webhook-server-cert" not found
Sep 17 19:15:25.678484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.678465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4f7974ad-93f1-4883-9f1a-43c708c6b4cb-data\") pod \"seaweedfs-64568bcd49-sr7zz\" (UID: \"4f7974ad-93f1-4883-9f1a-43c708c6b4cb\") " pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.680921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.680898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-cert\") pod \"kserve-controller-manager-66576f5db6-5gkql\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") " pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.688487 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.688074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pz6s\" (UniqueName: \"kubernetes.io/projected/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-kube-api-access-8pz6s\") pod \"kserve-controller-manager-66576f5db6-5gkql\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") " pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.692337 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.691239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjvk\" (UniqueName: \"kubernetes.io/projected/4f7974ad-93f1-4883-9f1a-43c708c6b4cb-kube-api-access-2sjvk\") pod \"seaweedfs-64568bcd49-sr7zz\" (UID: \"4f7974ad-93f1-4883-9f1a-43c708c6b4cb\") " pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.694878 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.694853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q489\" (UniqueName: \"kubernetes.io/projected/094fe571-f8e5-4841-8b88-feca944b2015-kube-api-access-7q489\") pod \"llmisvc-controller-manager-b64f7dcc6-7mn8m\" (UID: \"094fe571-f8e5-4841-8b88-feca944b2015\") " pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:25.713851 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.713831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:25.768160 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.768063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:25.985139 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:25.985110    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-5gkql"]
Sep 17 19:15:25.987658 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:15:25.987629    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29ea022_2c8f_4c8f_a2f4_12e03036f3c9.slice/crio-922b8fc18d6a2003fec28aa3a0aaf39e28594c986e34c5067e48a09b1eaa4224 WatchSource:0}: Error finding container 922b8fc18d6a2003fec28aa3a0aaf39e28594c986e34c5067e48a09b1eaa4224: Status 404 returned error can't find the container with id 922b8fc18d6a2003fec28aa3a0aaf39e28594c986e34c5067e48a09b1eaa4224
Sep 17 19:15:26.041218 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.041194    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sr7zz"]
Sep 17 19:15:26.042491 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:15:26.042466    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7974ad_93f1_4883_9f1a_43c708c6b4cb.slice/crio-34396bb8ee65177d2b12ead502240b22a8f2793c6ff5a52bf074636572d1e7d8 WatchSource:0}: Error finding container 34396bb8ee65177d2b12ead502240b22a8f2793c6ff5a52bf074636572d1e7d8: Status 404 returned error can't find the container with id 34396bb8ee65177d2b12ead502240b22a8f2793c6ff5a52bf074636572d1e7d8
Sep 17 19:15:26.183671 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.183645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094fe571-f8e5-4841-8b88-feca944b2015-cert\") pod \"llmisvc-controller-manager-b64f7dcc6-7mn8m\" (UID: \"094fe571-f8e5-4841-8b88-feca944b2015\") " pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:26.186038 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.186020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/094fe571-f8e5-4841-8b88-feca944b2015-cert\") pod \"llmisvc-controller-manager-b64f7dcc6-7mn8m\" (UID: \"094fe571-f8e5-4841-8b88-feca944b2015\") " pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:26.343455 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.343425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:26.618826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.617847    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"]
Sep 17 19:15:26.621118 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:15:26.621085    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod094fe571_f8e5_4841_8b88_feca944b2015.slice/crio-3c37480622275a775e6e3f555d03474f22a8a098669a4552892006ab659038e1 WatchSource:0}: Error finding container 3c37480622275a775e6e3f555d03474f22a8a098669a4552892006ab659038e1: Status 404 returned error can't find the container with id 3c37480622275a775e6e3f555d03474f22a8a098669a4552892006ab659038e1
Sep 17 19:15:26.767104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.767061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m" event={"ID":"094fe571-f8e5-4841-8b88-feca944b2015","Type":"ContainerStarted","Data":"3c37480622275a775e6e3f555d03474f22a8a098669a4552892006ab659038e1"}
Sep 17 19:15:26.769172 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.769141    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sr7zz" event={"ID":"4f7974ad-93f1-4883-9f1a-43c708c6b4cb","Type":"ContainerStarted","Data":"34396bb8ee65177d2b12ead502240b22a8f2793c6ff5a52bf074636572d1e7d8"}
Sep 17 19:15:26.771198 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:26.771143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-5gkql" event={"ID":"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9","Type":"ContainerStarted","Data":"922b8fc18d6a2003fec28aa3a0aaf39e28594c986e34c5067e48a09b1eaa4224"}
Sep 17 19:15:30.802149 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.802100    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-5gkql" event={"ID":"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9","Type":"ContainerStarted","Data":"2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf"}
Sep 17 19:15:30.802588 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.802234    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:15:30.804005 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.803979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m" event={"ID":"094fe571-f8e5-4841-8b88-feca944b2015","Type":"ContainerStarted","Data":"9754619509d6a83f7cb2b21557ab8828bcfd66a9b385a9b348fed3f268d66ee8"}
Sep 17 19:15:30.804141 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.804051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:15:30.805563 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.805542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sr7zz" event={"ID":"4f7974ad-93f1-4883-9f1a-43c708c6b4cb","Type":"ContainerStarted","Data":"4d5b1c48f586b3fac1942a5a91296066d3477ccb3c2597b39702c2dd3248d057"}
Sep 17 19:15:30.805672 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.805656    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:15:30.821227 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.821185    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66576f5db6-5gkql" podStartSLOduration=1.344078552 podStartE2EDuration="5.82117241s" totalImagesPullingTime="4.477093858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:30.820062695 +0000 UTC m=+532.867942555" watchObservedRunningTime="2026-09-17 19:15:30.82117241 +0000 UTC m=+532.869052266"
Sep 17 19:15:30.839334 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.839262    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m" podStartSLOduration=1.996833086 podStartE2EDuration="5.839247469s" totalImagesPullingTime="3.842414383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:30.836994747 +0000 UTC m=+532.884874604" watchObservedRunningTime="2026-09-17 19:15:30.839247469 +0000 UTC m=+532.887127333"
Sep 17 19:15:30.859482 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:30.859444    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-sr7zz" podStartSLOduration=1.343734964 podStartE2EDuration="5.859432465s" totalImagesPullingTime="4.515697501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:30.857994343 +0000 UTC m=+532.905874221" watchObservedRunningTime="2026-09-17 19:15:30.859432465 +0000 UTC m=+532.907312321"
Sep 17 19:15:36.813262 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:15:36.813229    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-sr7zz"
Sep 17 19:16:01.813243 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:01.813212    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-b64f7dcc6-7mn8m"
Sep 17 19:16:01.816271 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:01.816251    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:16:09.399857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.399821    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-5gkql"]
Sep 17 19:16:09.400406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.400029    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-66576f5db6-5gkql" podUID="d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" containerName="manager" containerID="cri-o://2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf" gracePeriod=10
Sep 17 19:16:09.431097 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.431069    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-6qjz6"]
Sep 17 19:16:09.433409 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.433393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.445274 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.445254    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-6qjz6"]
Sep 17 19:16:09.510560 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.510539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bnhd\" (UniqueName: \"kubernetes.io/projected/0dcc16c3-3650-4c98-b349-c52a27e4efba-kube-api-access-5bnhd\") pod \"kserve-controller-manager-66576f5db6-6qjz6\" (UID: \"0dcc16c3-3650-4c98-b349-c52a27e4efba\") " pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.510662 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.510596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dcc16c3-3650-4c98-b349-c52a27e4efba-cert\") pod \"kserve-controller-manager-66576f5db6-6qjz6\" (UID: \"0dcc16c3-3650-4c98-b349-c52a27e4efba\") " pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.611663 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.611627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dcc16c3-3650-4c98-b349-c52a27e4efba-cert\") pod \"kserve-controller-manager-66576f5db6-6qjz6\" (UID: \"0dcc16c3-3650-4c98-b349-c52a27e4efba\") " pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.611841 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.611724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bnhd\" (UniqueName: \"kubernetes.io/projected/0dcc16c3-3650-4c98-b349-c52a27e4efba-kube-api-access-5bnhd\") pod \"kserve-controller-manager-66576f5db6-6qjz6\" (UID: \"0dcc16c3-3650-4c98-b349-c52a27e4efba\") " pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.614033 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.614006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dcc16c3-3650-4c98-b349-c52a27e4efba-cert\") pod \"kserve-controller-manager-66576f5db6-6qjz6\" (UID: \"0dcc16c3-3650-4c98-b349-c52a27e4efba\") " pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.621439 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.621412    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bnhd\" (UniqueName: \"kubernetes.io/projected/0dcc16c3-3650-4c98-b349-c52a27e4efba-kube-api-access-5bnhd\") pod \"kserve-controller-manager-66576f5db6-6qjz6\" (UID: \"0dcc16c3-3650-4c98-b349-c52a27e4efba\") " pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.637769 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.637751    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:16:09.712622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.712558    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pz6s\" (UniqueName: \"kubernetes.io/projected/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-kube-api-access-8pz6s\") pod \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") "
Sep 17 19:16:09.712746 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.712637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-cert\") pod \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\" (UID: \"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9\") "
Sep 17 19:16:09.714761 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.714736    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-cert" (OuterVolumeSpecName: "cert") pod "d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" (UID: "d29ea022-2c8f-4c8f-a2f4-12e03036f3c9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:09.714860 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.714843    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-kube-api-access-8pz6s" (OuterVolumeSpecName: "kube-api-access-8pz6s") pod "d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" (UID: "d29ea022-2c8f-4c8f-a2f4-12e03036f3c9"). InnerVolumeSpecName "kube-api-access-8pz6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:09.773037 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.773012    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:09.814093 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.814061    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pz6s\" (UniqueName: \"kubernetes.io/projected/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-kube-api-access-8pz6s\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:16:09.814225 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:09.814097    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:16:10.025094 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.025015    2583 generic.go:356] "Generic (PLEG): container finished" podID="d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" containerID="2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf" exitCode=0
Sep 17 19:16:10.025094 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.025082    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-5gkql"
Sep 17 19:16:10.025331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.025108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-5gkql" event={"ID":"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9","Type":"ContainerDied","Data":"2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf"}
Sep 17 19:16:10.025331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.025198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-5gkql" event={"ID":"d29ea022-2c8f-4c8f-a2f4-12e03036f3c9","Type":"ContainerDied","Data":"922b8fc18d6a2003fec28aa3a0aaf39e28594c986e34c5067e48a09b1eaa4224"}
Sep 17 19:16:10.025331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.025230    2583 scope.go:117] "RemoveContainer" containerID="2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf"
Sep 17 19:16:10.034700 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.034673    2583 scope.go:117] "RemoveContainer" containerID="2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf"
Sep 17 19:16:10.035004 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:16:10.034985    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf\": container with ID starting with 2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf not found: ID does not exist" containerID="2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf"
Sep 17 19:16:10.035060 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.035009    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf"} err="failed to get container status \"2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf\": rpc error: code = NotFound desc = could not find container \"2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf\": container with ID starting with 2fcc82a187504a7b8f4a5a96c212da36e589567ac45c8321df33c5d77a4c8ccf not found: ID does not exist"
Sep 17 19:16:10.044820 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.044799    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-6qjz6"]
Sep 17 19:16:10.046128 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:16:10.046094    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcc16c3_3650_4c98_b349_c52a27e4efba.slice/crio-9ed9d8b7964b0c711361682c51f5255023bc97242c8d215bd58aa8f2ffa0a5e7 WatchSource:0}: Error finding container 9ed9d8b7964b0c711361682c51f5255023bc97242c8d215bd58aa8f2ffa0a5e7: Status 404 returned error can't find the container with id 9ed9d8b7964b0c711361682c51f5255023bc97242c8d215bd58aa8f2ffa0a5e7
Sep 17 19:16:10.048406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.048389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-5gkql"]
Sep 17 19:16:10.054350 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.054330    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-5gkql"]
Sep 17 19:16:10.254871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:10.254846    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" path="/var/lib/kubelet/pods/d29ea022-2c8f-4c8f-a2f4-12e03036f3c9/volumes"
Sep 17 19:16:11.032097 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:11.032061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-6qjz6" event={"ID":"0dcc16c3-3650-4c98-b349-c52a27e4efba","Type":"ContainerStarted","Data":"58a35f5e485cb5ddbcb45b3df105a52b9946ca4c5b5980086bbe5180266050b5"}
Sep 17 19:16:11.032556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:11.032113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-6qjz6" event={"ID":"0dcc16c3-3650-4c98-b349-c52a27e4efba","Type":"ContainerStarted","Data":"9ed9d8b7964b0c711361682c51f5255023bc97242c8d215bd58aa8f2ffa0a5e7"}
Sep 17 19:16:11.032556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:11.032167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:11.049999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:11.049954    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66576f5db6-6qjz6" podStartSLOduration=1.739982328 podStartE2EDuration="2.049941571s" totalImagesPullingTime="309.959243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:11.048060677 +0000 UTC m=+573.095940533" watchObservedRunningTime="2026-09-17 19:16:11.049941571 +0000 UTC m=+573.097821487"
Sep 17 19:16:42.043632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:42.043597    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66576f5db6-6qjz6"
Sep 17 19:16:43.303499 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.303470    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-m7tqm"]
Sep 17 19:16:43.303954 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.303936    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" containerName="manager"
Sep 17 19:16:43.304150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.303959    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" containerName="manager"
Sep 17 19:16:43.304150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.304045    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29ea022-2c8f-4c8f-a2f4-12e03036f3c9" containerName="manager"
Sep 17 19:16:43.306977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.306953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.309533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.309513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-fjrnv\""
Sep 17 19:16:43.309633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.309564    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 19:16:43.315931 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.315907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-m7tqm"]
Sep 17 19:16:43.470804 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.470717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr7j\" (UniqueName: \"kubernetes.io/projected/3127ea0c-484e-4134-8c0d-cc5cefaee075-kube-api-access-pmr7j\") pod \"model-serving-api-7b954ffc95-m7tqm\" (UID: \"3127ea0c-484e-4134-8c0d-cc5cefaee075\") " pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.470804 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.470789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3127ea0c-484e-4134-8c0d-cc5cefaee075-tls-certs\") pod \"model-serving-api-7b954ffc95-m7tqm\" (UID: \"3127ea0c-484e-4134-8c0d-cc5cefaee075\") " pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.571237 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.571182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr7j\" (UniqueName: \"kubernetes.io/projected/3127ea0c-484e-4134-8c0d-cc5cefaee075-kube-api-access-pmr7j\") pod \"model-serving-api-7b954ffc95-m7tqm\" (UID: \"3127ea0c-484e-4134-8c0d-cc5cefaee075\") " pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.571237 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.571222    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3127ea0c-484e-4134-8c0d-cc5cefaee075-tls-certs\") pod \"model-serving-api-7b954ffc95-m7tqm\" (UID: \"3127ea0c-484e-4134-8c0d-cc5cefaee075\") " pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.573748 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.573722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3127ea0c-484e-4134-8c0d-cc5cefaee075-tls-certs\") pod \"model-serving-api-7b954ffc95-m7tqm\" (UID: \"3127ea0c-484e-4134-8c0d-cc5cefaee075\") " pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.579105 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.579085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr7j\" (UniqueName: \"kubernetes.io/projected/3127ea0c-484e-4134-8c0d-cc5cefaee075-kube-api-access-pmr7j\") pod \"model-serving-api-7b954ffc95-m7tqm\" (UID: \"3127ea0c-484e-4134-8c0d-cc5cefaee075\") " pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.618224 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.618200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:43.886501 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:43.886378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-m7tqm"]
Sep 17 19:16:43.889226 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:16:43.889194    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3127ea0c_484e_4134_8c0d_cc5cefaee075.slice/crio-caa9d7820b5fb09bbbed6d9185bbd238496b9a533a1bdd703221e418e4016416 WatchSource:0}: Error finding container caa9d7820b5fb09bbbed6d9185bbd238496b9a533a1bdd703221e418e4016416: Status 404 returned error can't find the container with id caa9d7820b5fb09bbbed6d9185bbd238496b9a533a1bdd703221e418e4016416
Sep 17 19:16:44.218340 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:44.218264    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-m7tqm" event={"ID":"3127ea0c-484e-4134-8c0d-cc5cefaee075","Type":"ContainerStarted","Data":"caa9d7820b5fb09bbbed6d9185bbd238496b9a533a1bdd703221e418e4016416"}
Sep 17 19:16:47.241242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:47.241202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-m7tqm" event={"ID":"3127ea0c-484e-4134-8c0d-cc5cefaee075","Type":"ContainerStarted","Data":"0c872eea709bc6cfa4bd0fe1f1da086d57f495f98c9da6d5572a83598067f540"}
Sep 17 19:16:47.241656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:47.241389    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:47.260942 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:47.260890    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-m7tqm" podStartSLOduration=1.932404317 podStartE2EDuration="4.260875229s" totalImagesPullingTime="2.328470912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:47.259130409 +0000 UTC m=+609.307010268" watchObservedRunningTime="2026-09-17 19:16:47.260875229 +0000 UTC m=+609.308755086"
Sep 17 19:16:58.255889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:58.255814    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-m7tqm"
Sep 17 19:16:59.724302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.724264    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-8p82n"]
Sep 17 19:16:59.731749 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.731720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8p82n"
Sep 17 19:16:59.734808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.734781    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 19:16:59.737643 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.737615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8p82n"]
Sep 17 19:16:59.805846 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.805816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202f4200-5551-44ca-9493-ca52c5fe4c51-model-cache\") pod \"s3-init-8p82n\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") " pod="kserve/s3-init-8p82n"
Sep 17 19:16:59.805967 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.805946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h698w\" (UniqueName: \"kubernetes.io/projected/202f4200-5551-44ca-9493-ca52c5fe4c51-kube-api-access-h698w\") pod \"s3-init-8p82n\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") " pod="kserve/s3-init-8p82n"
Sep 17 19:16:59.906612 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.906583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h698w\" (UniqueName: \"kubernetes.io/projected/202f4200-5551-44ca-9493-ca52c5fe4c51-kube-api-access-h698w\") pod \"s3-init-8p82n\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") " pod="kserve/s3-init-8p82n"
Sep 17 19:16:59.906612 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.906613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202f4200-5551-44ca-9493-ca52c5fe4c51-model-cache\") pod \"s3-init-8p82n\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") " pod="kserve/s3-init-8p82n"
Sep 17 19:16:59.906946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.906930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202f4200-5551-44ca-9493-ca52c5fe4c51-model-cache\") pod \"s3-init-8p82n\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") " pod="kserve/s3-init-8p82n"
Sep 17 19:16:59.916300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:16:59.916279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h698w\" (UniqueName: \"kubernetes.io/projected/202f4200-5551-44ca-9493-ca52c5fe4c51-kube-api-access-h698w\") pod \"s3-init-8p82n\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") " pod="kserve/s3-init-8p82n"
Sep 17 19:17:00.042012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:00.041937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8p82n"
Sep 17 19:17:00.321409 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:00.321383    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8p82n"]
Sep 17 19:17:00.323184 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:17:00.323161    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod202f4200_5551_44ca_9493_ca52c5fe4c51.slice/crio-a9993c32fd5fdf6443882dd53a810c00fa62f8df451ae3ebf55d3123ab639d75 WatchSource:0}: Error finding container a9993c32fd5fdf6443882dd53a810c00fa62f8df451ae3ebf55d3123ab639d75: Status 404 returned error can't find the container with id a9993c32fd5fdf6443882dd53a810c00fa62f8df451ae3ebf55d3123ab639d75
Sep 17 19:17:01.317652 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:01.317613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8p82n" event={"ID":"202f4200-5551-44ca-9493-ca52c5fe4c51","Type":"ContainerStarted","Data":"a9993c32fd5fdf6443882dd53a810c00fa62f8df451ae3ebf55d3123ab639d75"}
Sep 17 19:17:04.337553 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:04.337523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8p82n" event={"ID":"202f4200-5551-44ca-9493-ca52c5fe4c51","Type":"ContainerStarted","Data":"0d50f1c6072c5a00fcfa3129fd3529edad749688203c0a2745e0393079336aa3"}
Sep 17 19:17:06.350431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:06.350394    2583 generic.go:356] "Generic (PLEG): container finished" podID="202f4200-5551-44ca-9493-ca52c5fe4c51" containerID="0d50f1c6072c5a00fcfa3129fd3529edad749688203c0a2745e0393079336aa3" exitCode=0
Sep 17 19:17:06.350815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:06.350459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8p82n" event={"ID":"202f4200-5551-44ca-9493-ca52c5fe4c51","Type":"ContainerDied","Data":"0d50f1c6072c5a00fcfa3129fd3529edad749688203c0a2745e0393079336aa3"}
Sep 17 19:17:11.387523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:11.387483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8p82n" event={"ID":"202f4200-5551-44ca-9493-ca52c5fe4c51","Type":"ContainerStarted","Data":"21387d625b4cc5d2c873e09149d786dd3489affea0a9479318d2608709210f1f"}
Sep 17 19:17:11.407920 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:17:11.407862    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-8p82n" podStartSLOduration=2.416898649 podStartE2EDuration="12.407843538s" totalImagesPullingTime="7.768607339s" totalInitContainerRuntime="2.22233755s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:11.404845728 +0000 UTC m=+633.452725588" watchObservedRunningTime="2026-09-17 19:17:11.407843538 +0000 UTC m=+633.455723587"
Sep 17 19:18:15.751148 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:15.751112    2583 generic.go:356] "Generic (PLEG): container finished" podID="202f4200-5551-44ca-9493-ca52c5fe4c51" containerID="21387d625b4cc5d2c873e09149d786dd3489affea0a9479318d2608709210f1f" exitCode=0
Sep 17 19:18:15.751542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:15.751187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8p82n" event={"ID":"202f4200-5551-44ca-9493-ca52c5fe4c51","Type":"ContainerDied","Data":"21387d625b4cc5d2c873e09149d786dd3489affea0a9479318d2608709210f1f"}
Sep 17 19:18:16.897090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:16.892192    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8p82n"
Sep 17 19:18:16.929993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:16.929962    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202f4200-5551-44ca-9493-ca52c5fe4c51-model-cache\") pod \"202f4200-5551-44ca-9493-ca52c5fe4c51\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") "
Sep 17 19:18:16.930138 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:16.930053    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h698w\" (UniqueName: \"kubernetes.io/projected/202f4200-5551-44ca-9493-ca52c5fe4c51-kube-api-access-h698w\") pod \"202f4200-5551-44ca-9493-ca52c5fe4c51\" (UID: \"202f4200-5551-44ca-9493-ca52c5fe4c51\") "
Sep 17 19:18:16.932152 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:16.932121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202f4200-5551-44ca-9493-ca52c5fe4c51-kube-api-access-h698w" (OuterVolumeSpecName: "kube-api-access-h698w") pod "202f4200-5551-44ca-9493-ca52c5fe4c51" (UID: "202f4200-5551-44ca-9493-ca52c5fe4c51"). InnerVolumeSpecName "kube-api-access-h698w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:18:16.948949 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:16.948922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202f4200-5551-44ca-9493-ca52c5fe4c51-model-cache" (OuterVolumeSpecName: "model-cache") pod "202f4200-5551-44ca-9493-ca52c5fe4c51" (UID: "202f4200-5551-44ca-9493-ca52c5fe4c51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:17.030960 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:17.030900    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202f4200-5551-44ca-9493-ca52c5fe4c51-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:18:17.030960 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:17.030925    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h698w\" (UniqueName: \"kubernetes.io/projected/202f4200-5551-44ca-9493-ca52c5fe4c51-kube-api-access-h698w\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:18:17.766319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:17.766293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8p82n"
Sep 17 19:18:17.766487 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:17.766326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8p82n" event={"ID":"202f4200-5551-44ca-9493-ca52c5fe4c51","Type":"ContainerDied","Data":"a9993c32fd5fdf6443882dd53a810c00fa62f8df451ae3ebf55d3123ab639d75"}
Sep 17 19:18:17.766487 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:17.766374    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9993c32fd5fdf6443882dd53a810c00fa62f8df451ae3ebf55d3123ab639d75"
Sep 17 19:18:42.878448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.878412    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"]
Sep 17 19:18:42.878866 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.878803    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202f4200-5551-44ca-9493-ca52c5fe4c51" containerName="s3-init"
Sep 17 19:18:42.878866 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.878816    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="202f4200-5551-44ca-9493-ca52c5fe4c51" containerName="s3-init"
Sep 17 19:18:42.878866 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.878830    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202f4200-5551-44ca-9493-ca52c5fe4c51" containerName="download-hf-model"
Sep 17 19:18:42.878866 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.878836    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="202f4200-5551-44ca-9493-ca52c5fe4c51" containerName="download-hf-model"
Sep 17 19:18:42.879033 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.878899    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="202f4200-5551-44ca-9493-ca52c5fe4c51" containerName="s3-init"
Sep 17 19:18:42.881105 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.881090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:42.885356 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.885332    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 19:18:42.885489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.885360    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-2dmbc\""
Sep 17 19:18:42.885489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.885360    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 19:18:42.885489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.885391    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 19:18:42.893757 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:42.893733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"]
Sep 17 19:18:43.037108 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.037084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.037228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.037128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.037228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.037209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.037317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.037249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.037317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.037271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.037317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.037296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftpp\" (UniqueName: \"kubernetes.io/projected/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-kube-api-access-gftpp\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138349 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138349 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138349 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138575 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftpp\" (UniqueName: \"kubernetes.io/projected/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-kube-api-access-gftpp\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138575 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138575 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138774 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138855 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.138918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.138903    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.140882 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.140861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.141022 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.141006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.148251 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.148228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftpp\" (UniqueName: \"kubernetes.io/projected/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-kube-api-access-gftpp\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.191397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.191372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:43.396483 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.396400    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"]
Sep 17 19:18:43.399031 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.399009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.401967 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.401949    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-wj8g4\""
Sep 17 19:18:43.414361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.414342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"]
Sep 17 19:18:43.476681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.476659    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"]
Sep 17 19:18:43.479698 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:18:43.479649    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c0e530_47d0_437f_8c5f_42f88d6cb3ba.slice/crio-18cf34cab2425d5c0dc9986a1c521ade5eb2405aba103c463ba0f1f58832ebc2 WatchSource:0}: Error finding container 18cf34cab2425d5c0dc9986a1c521ade5eb2405aba103c463ba0f1f58832ebc2: Status 404 returned error can't find the container with id 18cf34cab2425d5c0dc9986a1c521ade5eb2405aba103c463ba0f1f58832ebc2
Sep 17 19:18:43.481582 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.481568    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:18:43.543248 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.543223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3324f482-e041-4f69-927d-deecf4bec8d6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.543341 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.543256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zf4w\" (UniqueName: \"kubernetes.io/projected/3324f482-e041-4f69-927d-deecf4bec8d6-kube-api-access-6zf4w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.644482 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.644447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3324f482-e041-4f69-927d-deecf4bec8d6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.644590 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.644489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zf4w\" (UniqueName: \"kubernetes.io/projected/3324f482-e041-4f69-927d-deecf4bec8d6-kube-api-access-6zf4w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.647285 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.647239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3324f482-e041-4f69-927d-deecf4bec8d6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.653206 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.653187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zf4w\" (UniqueName: \"kubernetes.io/projected/3324f482-e041-4f69-927d-deecf4bec8d6-kube-api-access-6zf4w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.710672 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.710623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:43.910124 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:43.910036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" event={"ID":"44c0e530-47d0-437f-8c5f-42f88d6cb3ba","Type":"ContainerStarted","Data":"18cf34cab2425d5c0dc9986a1c521ade5eb2405aba103c463ba0f1f58832ebc2"}
Sep 17 19:18:44.042218 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:44.042188    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"]
Sep 17 19:18:44.072827 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:18:44.072791    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3324f482_e041_4f69_927d_deecf4bec8d6.slice/crio-9e4bd84fc78d9a1a91cf1b40c399829fb3749ea14b22f9b5eb51dd44eed810d6 WatchSource:0}: Error finding container 9e4bd84fc78d9a1a91cf1b40c399829fb3749ea14b22f9b5eb51dd44eed810d6: Status 404 returned error can't find the container with id 9e4bd84fc78d9a1a91cf1b40c399829fb3749ea14b22f9b5eb51dd44eed810d6
Sep 17 19:18:44.923822 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:44.923736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" event={"ID":"3324f482-e041-4f69-927d-deecf4bec8d6","Type":"ContainerStarted","Data":"9e4bd84fc78d9a1a91cf1b40c399829fb3749ea14b22f9b5eb51dd44eed810d6"}
Sep 17 19:18:45.932131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:45.932090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" event={"ID":"44c0e530-47d0-437f-8c5f-42f88d6cb3ba","Type":"ContainerStarted","Data":"d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311"}
Sep 17 19:18:45.952158 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:45.952088    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" podStartSLOduration=2.249781646 podStartE2EDuration="3.952072293s" totalImagesPullingTime="1.702290647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:45.951838086 +0000 UTC m=+727.999717946" watchObservedRunningTime="2026-09-17 19:18:45.952072293 +0000 UTC m=+727.999952151"
Sep 17 19:18:46.191637 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:46.191561    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:46.939324 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:46.939279    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" event={"ID":"3324f482-e041-4f69-927d-deecf4bec8d6","Type":"ContainerStarted","Data":"d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6"}
Sep 17 19:18:46.939848 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:46.939393    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:18:46.956981 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:46.956931    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podStartSLOduration=1.724204479 podStartE2EDuration="3.956915956s" totalImagesPullingTime="2.232711477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:46.955374582 +0000 UTC m=+729.003254439" watchObservedRunningTime="2026-09-17 19:18:46.956915956 +0000 UTC m=+729.004795813"
Sep 17 19:18:53.191515 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:53.191480    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:53.205000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:53.204973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:18:53.992954 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:18:53.992925    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:19:17.945860 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:17.945818    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:27.946239 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:27.946198    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:37.946384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:37.946346    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:41.640316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:41.640274    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"]
Sep 17 19:19:41.642198 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:41.642155    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:41.658230 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:41.658204    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"]
Sep 17 19:19:41.667267 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:19:41.667240    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:19:41.667267 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:19:41.667267 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:19:41.667267 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311" cmd=["/bin/sleep","15"]
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:41.667286    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" message="" err=<
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:19:41.667298    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" podUID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" containerName="main" containerID="cri-o://d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311"
Sep 17 19:19:41.667510 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:41.667331    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" podUID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" containerName="main" containerID="cri-o://d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311" gracePeriod=60
Sep 17 19:19:42.012341 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.012314    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:19:42.125872 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.125841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tls-certs\") pod \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") "
Sep 17 19:19:42.126041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.125885    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-model-cache\") pod \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") "
Sep 17 19:19:42.126041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.125903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tmp-dir\") pod \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") "
Sep 17 19:19:42.126041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.125951    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-dshm\") pod \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") "
Sep 17 19:19:42.126041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.125973    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-home\") pod \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") "
Sep 17 19:19:42.126041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.126015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gftpp\" (UniqueName: \"kubernetes.io/projected/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-kube-api-access-gftpp\") pod \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\" (UID: \"44c0e530-47d0-437f-8c5f-42f88d6cb3ba\") "
Sep 17 19:19:42.126321 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.126121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "44c0e530-47d0-437f-8c5f-42f88d6cb3ba" (UID: "44c0e530-47d0-437f-8c5f-42f88d6cb3ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:42.126321 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.126232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44c0e530-47d0-437f-8c5f-42f88d6cb3ba" (UID: "44c0e530-47d0-437f-8c5f-42f88d6cb3ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:42.126321 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.126261    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-home" (OuterVolumeSpecName: "home") pod "44c0e530-47d0-437f-8c5f-42f88d6cb3ba" (UID: "44c0e530-47d0-437f-8c5f-42f88d6cb3ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:42.126476 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.126339    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:19:42.126476 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.126358    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:19:42.126476 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.126395    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:19:42.128274 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.128247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-kube-api-access-gftpp" (OuterVolumeSpecName: "kube-api-access-gftpp") pod "44c0e530-47d0-437f-8c5f-42f88d6cb3ba" (UID: "44c0e530-47d0-437f-8c5f-42f88d6cb3ba"). InnerVolumeSpecName "kube-api-access-gftpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:42.128374 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.128285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44c0e530-47d0-437f-8c5f-42f88d6cb3ba" (UID: "44c0e530-47d0-437f-8c5f-42f88d6cb3ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:42.128374 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.128322    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-dshm" (OuterVolumeSpecName: "dshm") pod "44c0e530-47d0-437f-8c5f-42f88d6cb3ba" (UID: "44c0e530-47d0-437f-8c5f-42f88d6cb3ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:42.227756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.227678    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gftpp\" (UniqueName: \"kubernetes.io/projected/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-kube-api-access-gftpp\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:19:42.227756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.227721    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:19:42.227756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.227731    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44c0e530-47d0-437f-8c5f-42f88d6cb3ba-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:19:42.253092 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.253068    2583 generic.go:356] "Generic (PLEG): container finished" podID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" containerID="d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311" exitCode=0
Sep 17 19:19:42.253218 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.253183    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"
Sep 17 19:19:42.256391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.256370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" event={"ID":"44c0e530-47d0-437f-8c5f-42f88d6cb3ba","Type":"ContainerDied","Data":"d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311"}
Sep 17 19:19:42.256480 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.256445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" event={"ID":"44c0e530-47d0-437f-8c5f-42f88d6cb3ba","Type":"ContainerDied","Data":"18cf34cab2425d5c0dc9986a1c521ade5eb2405aba103c463ba0f1f58832ebc2"}
Sep 17 19:19:42.256480 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.256479    2583 scope.go:117] "RemoveContainer" containerID="d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311"
Sep 17 19:19:42.266431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.266414    2583 scope.go:117] "RemoveContainer" containerID="d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311"
Sep 17 19:19:42.266721 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:19:42.266678    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311\": container with ID starting with d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311 not found: ID does not exist" containerID="d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311"
Sep 17 19:19:42.266789 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.266730    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311"} err="failed to get container status \"d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311\": rpc error: code = NotFound desc = could not find container \"d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311\": container with ID starting with d4dc2fa3f64854c77badac99bf1c6dc49b6ec3f07e074a7818a92c7fba159311 not found: ID does not exist"
Sep 17 19:19:42.277878 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.277855    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"]
Sep 17 19:19:42.281852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.281826    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf"]
Sep 17 19:19:42.982468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:42.982430    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tvzrf" podUID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:19:44.255399 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:44.255366    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" path="/var/lib/kubelet/pods/44c0e530-47d0-437f-8c5f-42f88d6cb3ba/volumes"
Sep 17 19:19:51.642570 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:51.642483    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:56.662477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:19:56.662426    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" containerID="cri-o://d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6" gracePeriod=45
Sep 17 19:20:01.642706 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:01.642658    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:11.642233 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:11.642194    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:21.642642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:21.642604    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:20:26.906977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:26.906951    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:20:26.966236 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:26.966168    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3324f482-e041-4f69-927d-deecf4bec8d6-tls-certs\") pod \"3324f482-e041-4f69-927d-deecf4bec8d6\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") "
Sep 17 19:20:26.966344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:26.966283    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zf4w\" (UniqueName: \"kubernetes.io/projected/3324f482-e041-4f69-927d-deecf4bec8d6-kube-api-access-6zf4w\") pod \"3324f482-e041-4f69-927d-deecf4bec8d6\" (UID: \"3324f482-e041-4f69-927d-deecf4bec8d6\") "
Sep 17 19:20:26.968622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:26.968591    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3324f482-e041-4f69-927d-deecf4bec8d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3324f482-e041-4f69-927d-deecf4bec8d6" (UID: "3324f482-e041-4f69-927d-deecf4bec8d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:26.968767 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:26.968645    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3324f482-e041-4f69-927d-deecf4bec8d6-kube-api-access-6zf4w" (OuterVolumeSpecName: "kube-api-access-6zf4w") pod "3324f482-e041-4f69-927d-deecf4bec8d6" (UID: "3324f482-e041-4f69-927d-deecf4bec8d6"). InnerVolumeSpecName "kube-api-access-6zf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:27.067573 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.067548    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zf4w\" (UniqueName: \"kubernetes.io/projected/3324f482-e041-4f69-927d-deecf4bec8d6-kube-api-access-6zf4w\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:20:27.067573 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.067570    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3324f482-e041-4f69-927d-deecf4bec8d6-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:20:27.494632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.494600    2583 generic.go:356] "Generic (PLEG): container finished" podID="3324f482-e041-4f69-927d-deecf4bec8d6" containerID="d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6" exitCode=0
Sep 17 19:20:27.494886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.494642    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"
Sep 17 19:20:27.494886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.494668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" event={"ID":"3324f482-e041-4f69-927d-deecf4bec8d6","Type":"ContainerDied","Data":"d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6"}
Sep 17 19:20:27.494886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.494735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v" event={"ID":"3324f482-e041-4f69-927d-deecf4bec8d6","Type":"ContainerDied","Data":"9e4bd84fc78d9a1a91cf1b40c399829fb3749ea14b22f9b5eb51dd44eed810d6"}
Sep 17 19:20:27.494886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.494759    2583 scope.go:117] "RemoveContainer" containerID="d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6"
Sep 17 19:20:27.504132 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.504116    2583 scope.go:117] "RemoveContainer" containerID="d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6"
Sep 17 19:20:27.504419 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:20:27.504396    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6\": container with ID starting with d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6 not found: ID does not exist" containerID="d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6"
Sep 17 19:20:27.504504 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.504426    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6"} err="failed to get container status \"d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6\": rpc error: code = NotFound desc = could not find container \"d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6\": container with ID starting with d840c32eb82ca9f370233b56cd3a95f40aaa2814332872b94f335ee1321848c6 not found: ID does not exist"
Sep 17 19:20:27.516697 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.516657    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"]
Sep 17 19:20:27.520873 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:27.520851    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kjl4v"]
Sep 17 19:20:28.255336 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:28.255304    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" path="/var/lib/kubelet/pods/3324f482-e041-4f69-927d-deecf4bec8d6/volumes"
Sep 17 19:20:34.960144 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.960113    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"]
Sep 17 19:20:34.960521 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.960482    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" containerName="main"
Sep 17 19:20:34.960521 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.960492    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" containerName="main"
Sep 17 19:20:34.960521 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.960500    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main"
Sep 17 19:20:34.960521 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.960505    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main"
Sep 17 19:20:34.960679 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.960561    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3324f482-e041-4f69-927d-deecf4bec8d6" containerName="main"
Sep 17 19:20:34.960679 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.960570    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c0e530-47d0-437f-8c5f-42f88d6cb3ba" containerName="main"
Sep 17 19:20:34.965015 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.964994    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:34.969250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.969229    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-tpnbg\""
Sep 17 19:20:34.969356 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.969248    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 19:20:34.969849 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.969834    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 19:20:34.970025 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.970008    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 19:20:34.981736 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:34.981713    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"]
Sep 17 19:20:35.030012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.029985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svh4n\" (UniqueName: \"kubernetes.io/projected/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-kube-api-access-svh4n\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.030135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.030025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.030135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.030043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.030261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.030140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.030261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.030221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.030261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.030246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131447 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131570 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131570 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131570 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131570 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131570 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svh4n\" (UniqueName: \"kubernetes.io/projected/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-kube-api-access-svh4n\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131892 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.131956 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.132001 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.131989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.134071 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.134036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.134213 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.134196    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.139951 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.139931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svh4n\" (UniqueName: \"kubernetes.io/projected/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-kube-api-access-svh4n\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-h7pmc\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.275193 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.275125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:35.549680 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:35.549652    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"]
Sep 17 19:20:35.551147 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:20:35.551122    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2812cc51_8b7e_405f_9ccc_c9c50c0178f2.slice/crio-080db772fc1f430fb0bf734b65a032bfa1fc62d623db774be02a3833552776a3 WatchSource:0}: Error finding container 080db772fc1f430fb0bf734b65a032bfa1fc62d623db774be02a3833552776a3: Status 404 returned error can't find the container with id 080db772fc1f430fb0bf734b65a032bfa1fc62d623db774be02a3833552776a3
Sep 17 19:20:36.547158 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:36.547121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" event={"ID":"2812cc51-8b7e-405f-9ccc-c9c50c0178f2","Type":"ContainerStarted","Data":"b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398"}
Sep 17 19:20:36.547523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:36.547179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" event={"ID":"2812cc51-8b7e-405f-9ccc-c9c50c0178f2","Type":"ContainerStarted","Data":"080db772fc1f430fb0bf734b65a032bfa1fc62d623db774be02a3833552776a3"}
Sep 17 19:20:36.566152 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:36.566093    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" podStartSLOduration=2.566078606 podStartE2EDuration="2.566078606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:36.563874405 +0000 UTC m=+838.611754262" watchObservedRunningTime="2026-09-17 19:20:36.566078606 +0000 UTC m=+838.613958495"
Sep 17 19:20:37.276005 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:37.275979    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:45.275551 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:45.275519    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:45.288111 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:45.288085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:20:45.612371 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:20:45.612333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:21:02.457725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.457662    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"]
Sep 17 19:21:02.461037 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.461015    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.463439 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.463420    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 19:21:02.464810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.464786    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-22qhc\""
Sep 17 19:21:02.464917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.464835    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 19:21:02.464917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.464856    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 19:21:02.471178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.471160    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"]
Sep 17 19:21:02.550196 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.550168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.550317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.550200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hz5v\" (UniqueName: \"kubernetes.io/projected/37ff8f7d-c719-438e-93b7-1e742fd32f84-kube-api-access-5hz5v\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.550317 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.550228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.550437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.550370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-home\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.550437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.550403    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ff8f7d-c719-438e-93b7-1e742fd32f84-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.550545 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.550484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.651752 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.651730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-home\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.651862 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.651754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ff8f7d-c719-438e-93b7-1e742fd32f84-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.651862 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.651784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.651988 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.651954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.651988 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.651975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hz5v\" (UniqueName: \"kubernetes.io/projected/37ff8f7d-c719-438e-93b7-1e742fd32f84-kube-api-access-5hz5v\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.652099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.651997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.652194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.652173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-home\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.652372 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.652349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.652372 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.652357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.654213 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.654196    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.654378 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.654360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ff8f7d-c719-438e-93b7-1e742fd32f84-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.660957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.660940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hz5v\" (UniqueName: \"kubernetes.io/projected/37ff8f7d-c719-438e-93b7-1e742fd32f84-kube-api-access-5hz5v\") pod \"fc-concurrency-test-kserve-b5d44467c-jkpdb\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:02.772930 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:02.772866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:03.070497 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:03.070465    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"]
Sep 17 19:21:03.072155 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:21:03.072127    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ff8f7d_c719_438e_93b7_1e742fd32f84.slice/crio-176c56be5c2c8188860c481dc57a4d9bc159f95ca9a85860f15f85aebd5291f2 WatchSource:0}: Error finding container 176c56be5c2c8188860c481dc57a4d9bc159f95ca9a85860f15f85aebd5291f2: Status 404 returned error can't find the container with id 176c56be5c2c8188860c481dc57a4d9bc159f95ca9a85860f15f85aebd5291f2
Sep 17 19:21:03.697605 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:03.697566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" event={"ID":"37ff8f7d-c719-438e-93b7-1e742fd32f84","Type":"ContainerStarted","Data":"622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20"}
Sep 17 19:21:03.698065 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:03.697632    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" event={"ID":"37ff8f7d-c719-438e-93b7-1e742fd32f84","Type":"ContainerStarted","Data":"176c56be5c2c8188860c481dc57a4d9bc159f95ca9a85860f15f85aebd5291f2"}
Sep 17 19:21:03.720468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:03.720426    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" podStartSLOduration=1.720410776 podStartE2EDuration="1.720410776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:03.718046143 +0000 UTC m=+865.765926012" watchObservedRunningTime="2026-09-17 19:21:03.720410776 +0000 UTC m=+865.768290632"
Sep 17 19:21:03.773836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:03.773812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:12.774008 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:12.773971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:12.786400 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:12.786373    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:13.175303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.175262    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"]
Sep 17 19:21:13.184565 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:21:13.184527    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:21:13.184565 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:21:13.184565 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:21:13.184565 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398" cmd=["/bin/sleep","15"]
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.184582    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" message="" err=<
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:21:13.184594    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" podUID="2812cc51-8b7e-405f-9ccc-c9c50c0178f2" containerName="main" containerID="cri-o://b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398"
Sep 17 19:21:13.184850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.184638    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" podUID="2812cc51-8b7e-405f-9ccc-c9c50c0178f2" containerName="main" containerID="cri-o://b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398" gracePeriod=60
Sep 17 19:21:13.529063 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.529039    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:21:13.639983 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.639956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-model-cache\") pod \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") "
Sep 17 19:21:13.640135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svh4n\" (UniqueName: \"kubernetes.io/projected/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-kube-api-access-svh4n\") pod \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") "
Sep 17 19:21:13.640135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640084    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tmp-dir\") pod \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") "
Sep 17 19:21:13.640135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640133    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tls-certs\") pod \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") "
Sep 17 19:21:13.640303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-home\") pod \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") "
Sep 17 19:21:13.640303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640222    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-dshm\") pod \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\" (UID: \"2812cc51-8b7e-405f-9ccc-c9c50c0178f2\") "
Sep 17 19:21:13.640303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "2812cc51-8b7e-405f-9ccc-c9c50c0178f2" (UID: "2812cc51-8b7e-405f-9ccc-c9c50c0178f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:13.640463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640338    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2812cc51-8b7e-405f-9ccc-c9c50c0178f2" (UID: "2812cc51-8b7e-405f-9ccc-c9c50c0178f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:13.640463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640440    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-home" (OuterVolumeSpecName: "home") pod "2812cc51-8b7e-405f-9ccc-c9c50c0178f2" (UID: "2812cc51-8b7e-405f-9ccc-c9c50c0178f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:13.640575 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640556    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:13.640575 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640575    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:13.640673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.640583    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:13.642440 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.642414    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-dshm" (OuterVolumeSpecName: "dshm") pod "2812cc51-8b7e-405f-9ccc-c9c50c0178f2" (UID: "2812cc51-8b7e-405f-9ccc-c9c50c0178f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:13.642545 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.642487    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2812cc51-8b7e-405f-9ccc-c9c50c0178f2" (UID: "2812cc51-8b7e-405f-9ccc-c9c50c0178f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:13.642545 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.642508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-kube-api-access-svh4n" (OuterVolumeSpecName: "kube-api-access-svh4n") pod "2812cc51-8b7e-405f-9ccc-c9c50c0178f2" (UID: "2812cc51-8b7e-405f-9ccc-c9c50c0178f2"). InnerVolumeSpecName "kube-api-access-svh4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:13.741535 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.741473    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svh4n\" (UniqueName: \"kubernetes.io/projected/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-kube-api-access-svh4n\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:13.741535 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.741497    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:13.741535 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.741511    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2812cc51-8b7e-405f-9ccc-c9c50c0178f2-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:13.756031 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.756001    2583 generic.go:356] "Generic (PLEG): container finished" podID="2812cc51-8b7e-405f-9ccc-c9c50c0178f2" containerID="b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398" exitCode=0
Sep 17 19:21:13.756123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.756050    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"
Sep 17 19:21:13.756123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.756078    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" event={"ID":"2812cc51-8b7e-405f-9ccc-c9c50c0178f2","Type":"ContainerDied","Data":"b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398"}
Sep 17 19:21:13.756123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.756120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc" event={"ID":"2812cc51-8b7e-405f-9ccc-c9c50c0178f2","Type":"ContainerDied","Data":"080db772fc1f430fb0bf734b65a032bfa1fc62d623db774be02a3833552776a3"}
Sep 17 19:21:13.756242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.756141    2583 scope.go:117] "RemoveContainer" containerID="b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398"
Sep 17 19:21:13.765621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.765602    2583 scope.go:117] "RemoveContainer" containerID="b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398"
Sep 17 19:21:13.766032 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:21:13.765938    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398\": container with ID starting with b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398 not found: ID does not exist" containerID="b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398"
Sep 17 19:21:13.766032 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.765963    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398"} err="failed to get container status \"b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398\": rpc error: code = NotFound desc = could not find container \"b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398\": container with ID starting with b53880618555fa38e0253455da32701da374eed594134a32d517a0ade4c90398 not found: ID does not exist"
Sep 17 19:21:13.771053 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.771033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:13.780576 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.780557    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"]
Sep 17 19:21:13.788078 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:13.788056    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-h7pmc"]
Sep 17 19:21:14.254982 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:14.254945    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2812cc51-8b7e-405f-9ccc-c9c50c0178f2" path="/var/lib/kubelet/pods/2812cc51-8b7e-405f-9ccc-c9c50c0178f2/volumes"
Sep 17 19:21:41.301382 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.301342    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"]
Sep 17 19:21:41.315900 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:21:41.315856    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:21:41.315900 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:21:41.315900 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:21:41.315900 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20" cmd=["/bin/sleep","15"]
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.315910    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" message="" err=<
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:21:41.315922    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" podUID="37ff8f7d-c719-438e-93b7-1e742fd32f84" containerName="main" containerID="cri-o://622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20"
Sep 17 19:21:41.316215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.315964    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" podUID="37ff8f7d-c719-438e-93b7-1e742fd32f84" containerName="main" containerID="cri-o://622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20" gracePeriod=60
Sep 17 19:21:41.672131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.672103    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:41.747754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.747721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-tmp-dir\") pod \"37ff8f7d-c719-438e-93b7-1e742fd32f84\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") "
Sep 17 19:21:41.747893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.747764    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hz5v\" (UniqueName: \"kubernetes.io/projected/37ff8f7d-c719-438e-93b7-1e742fd32f84-kube-api-access-5hz5v\") pod \"37ff8f7d-c719-438e-93b7-1e742fd32f84\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") "
Sep 17 19:21:41.747893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.747793    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ff8f7d-c719-438e-93b7-1e742fd32f84-tls-certs\") pod \"37ff8f7d-c719-438e-93b7-1e742fd32f84\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") "
Sep 17 19:21:41.747893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.747828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-home\") pod \"37ff8f7d-c719-438e-93b7-1e742fd32f84\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") "
Sep 17 19:21:41.747893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.747842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-model-cache\") pod \"37ff8f7d-c719-438e-93b7-1e742fd32f84\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") "
Sep 17 19:21:41.748104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.747899    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-dshm\") pod \"37ff8f7d-c719-438e-93b7-1e742fd32f84\" (UID: \"37ff8f7d-c719-438e-93b7-1e742fd32f84\") "
Sep 17 19:21:41.748104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.747993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37ff8f7d-c719-438e-93b7-1e742fd32f84" (UID: "37ff8f7d-c719-438e-93b7-1e742fd32f84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:41.748104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.748083    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-home" (OuterVolumeSpecName: "home") pod "37ff8f7d-c719-438e-93b7-1e742fd32f84" (UID: "37ff8f7d-c719-438e-93b7-1e742fd32f84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:41.748238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.748147    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-model-cache" (OuterVolumeSpecName: "model-cache") pod "37ff8f7d-c719-438e-93b7-1e742fd32f84" (UID: "37ff8f7d-c719-438e-93b7-1e742fd32f84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:41.748238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.748202    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:41.748238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.748224    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:41.750147 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.750117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-dshm" (OuterVolumeSpecName: "dshm") pod "37ff8f7d-c719-438e-93b7-1e742fd32f84" (UID: "37ff8f7d-c719-438e-93b7-1e742fd32f84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:41.750251 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.750185    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ff8f7d-c719-438e-93b7-1e742fd32f84-kube-api-access-5hz5v" (OuterVolumeSpecName: "kube-api-access-5hz5v") pod "37ff8f7d-c719-438e-93b7-1e742fd32f84" (UID: "37ff8f7d-c719-438e-93b7-1e742fd32f84"). InnerVolumeSpecName "kube-api-access-5hz5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:41.750293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.750268    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ff8f7d-c719-438e-93b7-1e742fd32f84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37ff8f7d-c719-438e-93b7-1e742fd32f84" (UID: "37ff8f7d-c719-438e-93b7-1e742fd32f84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:41.848970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.848902    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:41.848970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.848924    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hz5v\" (UniqueName: \"kubernetes.io/projected/37ff8f7d-c719-438e-93b7-1e742fd32f84-kube-api-access-5hz5v\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:41.848970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.848936    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ff8f7d-c719-438e-93b7-1e742fd32f84-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:41.848970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.848950    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ff8f7d-c719-438e-93b7-1e742fd32f84-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:21:41.913459 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.913435    2583 generic.go:356] "Generic (PLEG): container finished" podID="37ff8f7d-c719-438e-93b7-1e742fd32f84" containerID="622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20" exitCode=0
Sep 17 19:21:41.913595 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.913482    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"
Sep 17 19:21:41.913595 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.913482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" event={"ID":"37ff8f7d-c719-438e-93b7-1e742fd32f84","Type":"ContainerDied","Data":"622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20"}
Sep 17 19:21:41.913728 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.913608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb" event={"ID":"37ff8f7d-c719-438e-93b7-1e742fd32f84","Type":"ContainerDied","Data":"176c56be5c2c8188860c481dc57a4d9bc159f95ca9a85860f15f85aebd5291f2"}
Sep 17 19:21:41.913728 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.913639    2583 scope.go:117] "RemoveContainer" containerID="622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20"
Sep 17 19:21:41.923422 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.923402    2583 scope.go:117] "RemoveContainer" containerID="622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20"
Sep 17 19:21:41.923721 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:21:41.923673    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20\": container with ID starting with 622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20 not found: ID does not exist" containerID="622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20"
Sep 17 19:21:41.923825 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.923730    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20"} err="failed to get container status \"622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20\": rpc error: code = NotFound desc = could not find container \"622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20\": container with ID starting with 622a128a5081fa3ab87cc4c7f31f116778558b5a5cde7e3a81d0bbd3c493fe20 not found: ID does not exist"
Sep 17 19:21:41.936365 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.936344    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"]
Sep 17 19:21:41.940161 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:41.940141    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jkpdb"]
Sep 17 19:21:42.254867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:21:42.254791    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ff8f7d-c719-438e-93b7-1e742fd32f84" path="/var/lib/kubelet/pods/37ff8f7d-c719-438e-93b7-1e742fd32f84/volumes"
Sep 17 19:22:09.433371 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.433338    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"]
Sep 17 19:22:09.435912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.433756    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ff8f7d-c719-438e-93b7-1e742fd32f84" containerName="main"
Sep 17 19:22:09.435912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.433768    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ff8f7d-c719-438e-93b7-1e742fd32f84" containerName="main"
Sep 17 19:22:09.435912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.433778    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2812cc51-8b7e-405f-9ccc-c9c50c0178f2" containerName="main"
Sep 17 19:22:09.435912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.433784    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2812cc51-8b7e-405f-9ccc-c9c50c0178f2" containerName="main"
Sep 17 19:22:09.435912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.433845    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ff8f7d-c719-438e-93b7-1e742fd32f84" containerName="main"
Sep 17 19:22:09.435912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.433852    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2812cc51-8b7e-405f-9ccc-c9c50c0178f2" containerName="main"
Sep 17 19:22:09.436932 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.436915    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.440796 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.440765    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 19:22:09.440937 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.440857    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 19:22:09.440937 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.440889    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-cch6v\""
Sep 17 19:22:09.441059 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.440931    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-4cqpv\""
Sep 17 19:22:09.441059 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.440935    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 19:22:09.450174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.450152    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"]
Sep 17 19:22:09.558606 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.558579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.558778 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.558633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.558778 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.558726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.558778 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.558754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.558893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.558780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c130a3-0d8d-472b-8390-c063d83b367b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.558893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.558821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72z5v\" (UniqueName: \"kubernetes.io/projected/62c130a3-0d8d-472b-8390-c063d83b367b-kube-api-access-72z5v\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660203 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c130a3-0d8d-472b-8390-c063d83b367b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72z5v\" (UniqueName: \"kubernetes.io/projected/62c130a3-0d8d-472b-8390-c063d83b367b-kube-api-access-72z5v\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.660944 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.660908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.662761 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.662739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.663115 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.663094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c130a3-0d8d-472b-8390-c063d83b367b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.670665 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.670619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72z5v\" (UniqueName: \"kubernetes.io/projected/62c130a3-0d8d-472b-8390-c063d83b367b-kube-api-access-72z5v\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:09.747823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:09.747764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:10.028300 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:10.028269    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"]
Sep 17 19:22:10.029588 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:22:10.029564    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c130a3_0d8d_472b_8390_c063d83b367b.slice/crio-83fc2a9b3cf2e41d3c11296096778fd577da17e3b8bcfadb86fb1e59f72ae617 WatchSource:0}: Error finding container 83fc2a9b3cf2e41d3c11296096778fd577da17e3b8bcfadb86fb1e59f72ae617: Status 404 returned error can't find the container with id 83fc2a9b3cf2e41d3c11296096778fd577da17e3b8bcfadb86fb1e59f72ae617
Sep 17 19:22:10.065673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:10.065646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" event={"ID":"62c130a3-0d8d-472b-8390-c063d83b367b","Type":"ContainerStarted","Data":"83fc2a9b3cf2e41d3c11296096778fd577da17e3b8bcfadb86fb1e59f72ae617"}
Sep 17 19:22:12.077504 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:12.077465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" event={"ID":"62c130a3-0d8d-472b-8390-c063d83b367b","Type":"ContainerStarted","Data":"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"}
Sep 17 19:22:12.077891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:12.077529    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:13.084931 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:13.084889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" event={"ID":"62c130a3-0d8d-472b-8390-c063d83b367b","Type":"ContainerStarted","Data":"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"}
Sep 17 19:22:13.114384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:13.114337    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" podStartSLOduration=2.686701774 podStartE2EDuration="4.114321335s" totalImagesPullingTime="1.427619561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:22:13.110351451 +0000 UTC m=+935.158231307" watchObservedRunningTime="2026-09-17 19:22:13.114321335 +0000 UTC m=+935.162201191"
Sep 17 19:22:13.748796 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:13.748745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:19.748306 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:19.748267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:19.761041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:19.761018    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:20.133811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:20.133784    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:21.141043 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:21.141015    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:22:31.596967 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.596922    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"]
Sep 17 19:22:31.601255 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.601232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.604293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.604267    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 19:22:31.604406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.604267    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-4f62p\""
Sep 17 19:22:31.604406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.604386    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:22:31.604547 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.604447    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 19:22:31.616098 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.616072    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"]
Sep 17 19:22:31.740918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.740888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnjp\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-kube-api-access-qvnjp\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741058 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.740924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741058 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.740984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741058 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.741009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741058 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.741036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741058 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.741053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.741151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.741181    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.741228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.741201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.841886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.841862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842021 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.841891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842021 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.841911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842021 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.841930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842021 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.841975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842179 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnjp\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-kube-api-access-qvnjp\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842179 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842179 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842179 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842383 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842383 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842558 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.842901 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.842799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.843266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.843244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.844445 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.844424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.844643 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.844624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.850614 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.850559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnjp\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-kube-api-access-qvnjp\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.850726 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.850618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-9dmwn\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:31.912721 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:31.912667    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:32.190843 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:32.190812    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"]
Sep 17 19:22:32.191516 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:22:32.191489    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f7eb96_3313_4a02_a4ce_64dfe76fbb69.slice/crio-1b181be1b1fb889630d2f5a4490cd77cd6e1d8c9f3181bb31bef5ab9f596966a WatchSource:0}: Error finding container 1b181be1b1fb889630d2f5a4490cd77cd6e1d8c9f3181bb31bef5ab9f596966a: Status 404 returned error can't find the container with id 1b181be1b1fb889630d2f5a4490cd77cd6e1d8c9f3181bb31bef5ab9f596966a
Sep 17 19:22:33.194055 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:33.194017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn" event={"ID":"46f7eb96-3313-4a02-a4ce-64dfe76fbb69","Type":"ContainerStarted","Data":"1b181be1b1fb889630d2f5a4490cd77cd6e1d8c9f3181bb31bef5ab9f596966a"}
Sep 17 19:22:35.129833 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:35.129796    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 17 19:22:35.130079 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:35.129873    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 17 19:22:35.130079 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:35.129902    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 17 19:22:35.207992 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:35.207959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn" event={"ID":"46f7eb96-3313-4a02-a4ce-64dfe76fbb69","Type":"ContainerStarted","Data":"96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1"}
Sep 17 19:22:35.232144 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:35.232091    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn" podStartSLOduration=1.296173961 podStartE2EDuration="4.232070231s" totalImagesPullingTime="2.93589627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:22:35.228851558 +0000 UTC m=+957.276731417" watchObservedRunningTime="2026-09-17 19:22:35.232070231 +0000 UTC m=+957.279950090"
Sep 17 19:22:35.913443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:35.913412    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:36.918638 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:36.918606    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:37.218822 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:37.218755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:37.219628 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:37.219611    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:45.854397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.854358    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"]
Sep 17 19:22:45.857814 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.857794    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:45.861710 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.861658    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-5q9zm\""
Sep 17 19:22:45.861710 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.861699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 19:22:45.867598 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.867576    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"]
Sep 17 19:22:45.955437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.955400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:45.955600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.955443    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e33097-7718-4669-90a6-1ac1c64c631c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:45.955600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.955491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:45.955600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.955552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:45.955806 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.955611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bnw\" (UniqueName: \"kubernetes.io/projected/f3e33097-7718-4669-90a6-1ac1c64c631c-kube-api-access-c6bnw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:45.955806 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:45.955717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.056996 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.056964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057130 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057190 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bnw\" (UniqueName: \"kubernetes.io/projected/f3e33097-7718-4669-90a6-1ac1c64c631c-kube-api-access-c6bnw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057304 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057356 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e33097-7718-4669-90a6-1ac1c64c631c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057356 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057303    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057470 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.057558 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.057542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.059576 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.059551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.059845 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.059829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e33097-7718-4669-90a6-1ac1c64c631c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.065818 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.065793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bnw\" (UniqueName: \"kubernetes.io/projected/f3e33097-7718-4669-90a6-1ac1c64c631c-kube-api-access-c6bnw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.169681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.169623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:46.446859 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:46.446830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"]
Sep 17 19:22:46.448991 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:22:46.448965    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e33097_7718_4669_90a6_1ac1c64c631c.slice/crio-63788924a018535309eb617a8dd39ba0ee89c61ddab74cf27cc33af46de7bf78 WatchSource:0}: Error finding container 63788924a018535309eb617a8dd39ba0ee89c61ddab74cf27cc33af46de7bf78: Status 404 returned error can't find the container with id 63788924a018535309eb617a8dd39ba0ee89c61ddab74cf27cc33af46de7bf78
Sep 17 19:22:47.274893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:47.274858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" event={"ID":"f3e33097-7718-4669-90a6-1ac1c64c631c","Type":"ContainerStarted","Data":"7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299"}
Sep 17 19:22:47.275249 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:47.274915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" event={"ID":"f3e33097-7718-4669-90a6-1ac1c64c631c","Type":"ContainerStarted","Data":"63788924a018535309eb617a8dd39ba0ee89c61ddab74cf27cc33af46de7bf78"}
Sep 17 19:22:47.296022 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:47.295979    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" podStartSLOduration=2.295964019 podStartE2EDuration="2.295964019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:22:47.293341472 +0000 UTC m=+969.341221427" watchObservedRunningTime="2026-09-17 19:22:47.295964019 +0000 UTC m=+969.343843876"
Sep 17 19:22:48.170482 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:48.170448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:53.007770 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.007737    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"]
Sep 17 19:22:53.014362 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.014337    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"]
Sep 17 19:22:53.014556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.014535    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn" podUID="46f7eb96-3313-4a02-a4ce-64dfe76fbb69" containerName="istio-proxy" containerID="cri-o://96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1" gracePeriod=30
Sep 17 19:22:53.017486 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:22:53.017447    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:22:53.017486 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:22:53.017486 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:22:53.017486 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299" cmd=["/bin/sleep","15"]
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.017511    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" message="" err=<
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:22:53.017529    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" podUID="f3e33097-7718-4669-90a6-1ac1c64c631c" containerName="main" containerID="cri-o://7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299"
Sep 17 19:22:53.017752 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.017573    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" podUID="f3e33097-7718-4669-90a6-1ac1c64c631c" containerName="main" containerID="cri-o://7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299" gracePeriod=60
Sep 17 19:22:53.268477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.268452    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:53.322294 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.322261    2583 generic.go:356] "Generic (PLEG): container finished" podID="f3e33097-7718-4669-90a6-1ac1c64c631c" containerID="7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299" exitCode=0
Sep 17 19:22:53.322463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.322299    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"
Sep 17 19:22:53.322463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.322362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" event={"ID":"f3e33097-7718-4669-90a6-1ac1c64c631c","Type":"ContainerDied","Data":"7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299"}
Sep 17 19:22:53.322463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.322406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5" event={"ID":"f3e33097-7718-4669-90a6-1ac1c64c631c","Type":"ContainerDied","Data":"63788924a018535309eb617a8dd39ba0ee89c61ddab74cf27cc33af46de7bf78"}
Sep 17 19:22:53.322463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.322434    2583 scope.go:117] "RemoveContainer" containerID="7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299"
Sep 17 19:22:53.326409 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326371    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e33097-7718-4669-90a6-1ac1c64c631c-tls-certs\") pod \"f3e33097-7718-4669-90a6-1ac1c64c631c\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") "
Sep 17 19:22:53.326527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326435    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-tmp-dir\") pod \"f3e33097-7718-4669-90a6-1ac1c64c631c\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") "
Sep 17 19:22:53.326527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326506    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-dshm\") pod \"f3e33097-7718-4669-90a6-1ac1c64c631c\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") "
Sep 17 19:22:53.326644 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326557    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6bnw\" (UniqueName: \"kubernetes.io/projected/f3e33097-7718-4669-90a6-1ac1c64c631c-kube-api-access-c6bnw\") pod \"f3e33097-7718-4669-90a6-1ac1c64c631c\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") "
Sep 17 19:22:53.326644 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-home\") pod \"f3e33097-7718-4669-90a6-1ac1c64c631c\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") "
Sep 17 19:22:53.326773 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326661    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-model-cache\") pod \"f3e33097-7718-4669-90a6-1ac1c64c631c\" (UID: \"f3e33097-7718-4669-90a6-1ac1c64c631c\") "
Sep 17 19:22:53.326773 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326712    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3e33097-7718-4669-90a6-1ac1c64c631c" (UID: "f3e33097-7718-4669-90a6-1ac1c64c631c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:53.327010 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-home" (OuterVolumeSpecName: "home") pod "f3e33097-7718-4669-90a6-1ac1c64c631c" (UID: "f3e33097-7718-4669-90a6-1ac1c64c631c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:53.327010 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.326995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3e33097-7718-4669-90a6-1ac1c64c631c" (UID: "f3e33097-7718-4669-90a6-1ac1c64c631c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:53.327189 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.327020    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:53.327189 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.327040    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:53.328920 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.328893    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e33097-7718-4669-90a6-1ac1c64c631c-kube-api-access-c6bnw" (OuterVolumeSpecName: "kube-api-access-c6bnw") pod "f3e33097-7718-4669-90a6-1ac1c64c631c" (UID: "f3e33097-7718-4669-90a6-1ac1c64c631c"). InnerVolumeSpecName "kube-api-access-c6bnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:53.329041 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.329020    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e33097-7718-4669-90a6-1ac1c64c631c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3e33097-7718-4669-90a6-1ac1c64c631c" (UID: "f3e33097-7718-4669-90a6-1ac1c64c631c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:53.329297 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.329280    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-dshm" (OuterVolumeSpecName: "dshm") pod "f3e33097-7718-4669-90a6-1ac1c64c631c" (UID: "f3e33097-7718-4669-90a6-1ac1c64c631c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:53.333010 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.332991    2583 scope.go:117] "RemoveContainer" containerID="7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299"
Sep 17 19:22:53.333275 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:22:53.333257    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299\": container with ID starting with 7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299 not found: ID does not exist" containerID="7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299"
Sep 17 19:22:53.333340 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.333287    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299"} err="failed to get container status \"7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299\": rpc error: code = NotFound desc = could not find container \"7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299\": container with ID starting with 7f31b01c7d00555963b5057973dae4f2fb14d35368af5b882fb651b04948d299 not found: ID does not exist"
Sep 17 19:22:53.428437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.428355    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:53.428437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.428394    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6bnw\" (UniqueName: \"kubernetes.io/projected/f3e33097-7718-4669-90a6-1ac1c64c631c-kube-api-access-c6bnw\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:53.428437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.428409    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e33097-7718-4669-90a6-1ac1c64c631c-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:53.428437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.428420    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e33097-7718-4669-90a6-1ac1c64c631c-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:53.648950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.648912    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"]
Sep 17 19:22:53.653000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:53.652972    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-654c5"]
Sep 17 19:22:54.256714 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:54.256664    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3e33097-7718-4669-90a6-1ac1c64c631c" path="/var/lib/kubelet/pods/f3e33097-7718-4669-90a6-1ac1c64c631c/volumes"
Sep 17 19:22:56.157665 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.157640    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:56.252770 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.252742    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvnjp\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-kube-api-access-qvnjp\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.252914 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.252780    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istiod-ca-cert\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.252914 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.252806    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-envoy\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.252914 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.252863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-credential-socket\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.252914 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.252879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-certs\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.252914 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.252911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-data\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.253246 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.252964    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-token\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.253246 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253047    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-podinfo\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.253246 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253080    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-socket\") pod \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\" (UID: \"46f7eb96-3313-4a02-a4ce-64dfe76fbb69\") "
Sep 17 19:22:56.253246 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253105    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:56.253246 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253239    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-data" (OuterVolumeSpecName: "istio-data") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:56.253505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:56.253505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253370    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-data\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.253505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253413    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istiod-ca-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.253505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253427    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-credential-socket\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.253505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253456    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:56.253967 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.253595    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:56.255258 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.255227    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-kube-api-access-qvnjp" (OuterVolumeSpecName: "kube-api-access-qvnjp") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "kube-api-access-qvnjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:56.255429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.255404    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:56.255528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.255501    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:22:56.255640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.255623    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-token" (OuterVolumeSpecName: "istio-token") pod "46f7eb96-3313-4a02-a4ce-64dfe76fbb69" (UID: "46f7eb96-3313-4a02-a4ce-64dfe76fbb69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:56.342528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.342500    2583 generic.go:356] "Generic (PLEG): container finished" podID="46f7eb96-3313-4a02-a4ce-64dfe76fbb69" containerID="96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1" exitCode=0
Sep 17 19:22:56.342630 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.342547    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"
Sep 17 19:22:56.342678 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.342542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn" event={"ID":"46f7eb96-3313-4a02-a4ce-64dfe76fbb69","Type":"ContainerDied","Data":"96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1"}
Sep 17 19:22:56.342678 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.342671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn" event={"ID":"46f7eb96-3313-4a02-a4ce-64dfe76fbb69","Type":"ContainerDied","Data":"1b181be1b1fb889630d2f5a4490cd77cd6e1d8c9f3181bb31bef5ab9f596966a"}
Sep 17 19:22:56.342777 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.342722    2583 scope.go:117] "RemoveContainer" containerID="96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1"
Sep 17 19:22:56.351506 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.351489    2583 scope.go:117] "RemoveContainer" containerID="96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1"
Sep 17 19:22:56.351833 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:22:56.351803    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1\": container with ID starting with 96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1 not found: ID does not exist" containerID="96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1"
Sep 17 19:22:56.351913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.351839    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1"} err="failed to get container status \"96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1\": rpc error: code = NotFound desc = could not find container \"96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1\": container with ID starting with 96322bd9d4e343d610a02ee74aaa541eadd3ee78a80671fa4cc616d5b3775eb1 not found: ID does not exist"
Sep 17 19:22:56.354240 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.354225    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-token\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.354287 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.354242    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-podinfo\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.354287 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.354251    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-socket\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.354287 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.354260    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvnjp\" (UniqueName: \"kubernetes.io/projected/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-kube-api-access-qvnjp\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.354287 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.354269    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-istio-envoy\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.354287 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.354277    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7eb96-3313-4a02-a4ce-64dfe76fbb69-workload-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:22:56.360612 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.360590    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"]
Sep 17 19:22:56.364780 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:56.364761    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-9dmwn"]
Sep 17 19:22:58.255319 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:22:58.255291    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f7eb96-3313-4a02-a4ce-64dfe76fbb69" path="/var/lib/kubelet/pods/46f7eb96-3313-4a02-a4ce-64dfe76fbb69/volumes"
Sep 17 19:23:06.365929 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.365894    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"]
Sep 17 19:23:06.366310 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.366296    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3e33097-7718-4669-90a6-1ac1c64c631c" containerName="main"
Sep 17 19:23:06.366365 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.366314    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3e33097-7718-4669-90a6-1ac1c64c631c" containerName="main"
Sep 17 19:23:06.366365 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.366337    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f7eb96-3313-4a02-a4ce-64dfe76fbb69" containerName="istio-proxy"
Sep 17 19:23:06.366365 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.366345    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f7eb96-3313-4a02-a4ce-64dfe76fbb69" containerName="istio-proxy"
Sep 17 19:23:06.366485 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.366444    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f7eb96-3313-4a02-a4ce-64dfe76fbb69" containerName="istio-proxy"
Sep 17 19:23:06.366485 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.366461    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3e33097-7718-4669-90a6-1ac1c64c631c" containerName="main"
Sep 17 19:23:06.371603 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.371582    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.374159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.374136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 19:23:06.374281 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.374175    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 19:23:06.375262 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.375243    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 19:23:06.375330 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.375284    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-jwn86\""
Sep 17 19:23:06.379929 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.379906    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"]
Sep 17 19:23:06.436834 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.436807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.436977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.436843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.436977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.436943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4htn\" (UniqueName: \"kubernetes.io/projected/a865d6f9-3428-42cb-ab48-fb41e1d45a31-kube-api-access-l4htn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.436977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.436973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.437104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.437002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.437104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.437034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.537844 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.537815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.537989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.537857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.537989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.537883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.538135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.538008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.538135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.538058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.538239 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.538162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4htn\" (UniqueName: \"kubernetes.io/projected/a865d6f9-3428-42cb-ab48-fb41e1d45a31-kube-api-access-l4htn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.538239 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.538193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.538343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.538328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.538398 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.538383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.540283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.540255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.540398 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.540326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.549808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.549783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4htn\" (UniqueName: \"kubernetes.io/projected/a865d6f9-3428-42cb-ab48-fb41e1d45a31-kube-api-access-l4htn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.683323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.683259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:06.972003 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:06.971924    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"]
Sep 17 19:23:06.976439 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:23:06.976405    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda865d6f9_3428_42cb_ab48_fb41e1d45a31.slice/crio-7d910eb03e241ff4cb1df839077365fd0e639e2d417b13338c9c96bf4f1296c0 WatchSource:0}: Error finding container 7d910eb03e241ff4cb1df839077365fd0e639e2d417b13338c9c96bf4f1296c0: Status 404 returned error can't find the container with id 7d910eb03e241ff4cb1df839077365fd0e639e2d417b13338c9c96bf4f1296c0
Sep 17 19:23:07.407639 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:07.407604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" event={"ID":"a865d6f9-3428-42cb-ab48-fb41e1d45a31","Type":"ContainerStarted","Data":"92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae"}
Sep 17 19:23:07.408023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:07.407665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" event={"ID":"a865d6f9-3428-42cb-ab48-fb41e1d45a31","Type":"ContainerStarted","Data":"7d910eb03e241ff4cb1df839077365fd0e639e2d417b13338c9c96bf4f1296c0"}
Sep 17 19:23:07.427858 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:07.427813    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" podStartSLOduration=1.4277965240000001 podStartE2EDuration="1.427796524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:07.426028342 +0000 UTC m=+989.473908212" watchObservedRunningTime="2026-09-17 19:23:07.427796524 +0000 UTC m=+989.475676381"
Sep 17 19:23:07.684269 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:07.684197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:08.588316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:08.588284    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"]
Sep 17 19:23:08.601758 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:23:08.601680    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:23:08.601758 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:08.601758 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:08.601758 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351" cmd=["/bin/sleep","15"]
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:08.601783    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" message="" err=<
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:23:08.601797    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="main" containerID="cri-o://93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"
Sep 17 19:23:08.602081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:08.601836    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="main" containerID="cri-o://93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351" gracePeriod=60
Sep 17 19:23:08.818318 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:08.818266    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="llm-d-routing-sidecar" containerID="cri-o://a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc" gracePeriod=60
Sep 17 19:23:09.064446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.064420    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:23:09.158978 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.158950    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-tmp-dir\") pod \"62c130a3-0d8d-472b-8390-c063d83b367b\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") "
Sep 17 19:23:09.159137 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c130a3-0d8d-472b-8390-c063d83b367b-tls-certs\") pod \"62c130a3-0d8d-472b-8390-c063d83b367b\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") "
Sep 17 19:23:09.159137 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72z5v\" (UniqueName: \"kubernetes.io/projected/62c130a3-0d8d-472b-8390-c063d83b367b-kube-api-access-72z5v\") pod \"62c130a3-0d8d-472b-8390-c063d83b367b\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") "
Sep 17 19:23:09.159137 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159104    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-home\") pod \"62c130a3-0d8d-472b-8390-c063d83b367b\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") "
Sep 17 19:23:09.159347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159148    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-dshm\") pod \"62c130a3-0d8d-472b-8390-c063d83b367b\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") "
Sep 17 19:23:09.159347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159192    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-model-cache\") pod \"62c130a3-0d8d-472b-8390-c063d83b367b\" (UID: \"62c130a3-0d8d-472b-8390-c063d83b367b\") "
Sep 17 19:23:09.159347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159242    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62c130a3-0d8d-472b-8390-c063d83b367b" (UID: "62c130a3-0d8d-472b-8390-c063d83b367b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:09.159512 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159374    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-home" (OuterVolumeSpecName: "home") pod "62c130a3-0d8d-472b-8390-c063d83b367b" (UID: "62c130a3-0d8d-472b-8390-c063d83b367b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:09.159512 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159485    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.159512 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159503    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.159636 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.159508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-model-cache" (OuterVolumeSpecName: "model-cache") pod "62c130a3-0d8d-472b-8390-c063d83b367b" (UID: "62c130a3-0d8d-472b-8390-c063d83b367b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:09.161616 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.161586    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c130a3-0d8d-472b-8390-c063d83b367b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62c130a3-0d8d-472b-8390-c063d83b367b" (UID: "62c130a3-0d8d-472b-8390-c063d83b367b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:09.161616 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.161603    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c130a3-0d8d-472b-8390-c063d83b367b-kube-api-access-72z5v" (OuterVolumeSpecName: "kube-api-access-72z5v") pod "62c130a3-0d8d-472b-8390-c063d83b367b" (UID: "62c130a3-0d8d-472b-8390-c063d83b367b"). InnerVolumeSpecName "kube-api-access-72z5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:09.161800 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.161655    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-dshm" (OuterVolumeSpecName: "dshm") pod "62c130a3-0d8d-472b-8390-c063d83b367b" (UID: "62c130a3-0d8d-472b-8390-c063d83b367b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:09.260121 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.260060    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.260121 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.260081    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62c130a3-0d8d-472b-8390-c063d83b367b-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.260121 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.260092    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c130a3-0d8d-472b-8390-c063d83b367b-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.260121 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.260100    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72z5v\" (UniqueName: \"kubernetes.io/projected/62c130a3-0d8d-472b-8390-c063d83b367b-kube-api-access-72z5v\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:09.420631 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.420603    2583 generic.go:356] "Generic (PLEG): container finished" podID="62c130a3-0d8d-472b-8390-c063d83b367b" containerID="93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351" exitCode=0
Sep 17 19:23:09.420631 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.420624    2583 generic.go:356] "Generic (PLEG): container finished" podID="62c130a3-0d8d-472b-8390-c063d83b367b" containerID="a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc" exitCode=0
Sep 17 19:23:09.420815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.420648    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"
Sep 17 19:23:09.420815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.420697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" event={"ID":"62c130a3-0d8d-472b-8390-c063d83b367b","Type":"ContainerDied","Data":"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"}
Sep 17 19:23:09.420815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.420757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" event={"ID":"62c130a3-0d8d-472b-8390-c063d83b367b","Type":"ContainerDied","Data":"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"}
Sep 17 19:23:09.420815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.420773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb" event={"ID":"62c130a3-0d8d-472b-8390-c063d83b367b","Type":"ContainerDied","Data":"83fc2a9b3cf2e41d3c11296096778fd577da17e3b8bcfadb86fb1e59f72ae617"}
Sep 17 19:23:09.420815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.420793    2583 scope.go:117] "RemoveContainer" containerID="93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"
Sep 17 19:23:09.431201 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.431169    2583 scope.go:117] "RemoveContainer" containerID="a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"
Sep 17 19:23:09.439222 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.439206    2583 scope.go:117] "RemoveContainer" containerID="93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"
Sep 17 19:23:09.439494 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:23:09.439473    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351\": container with ID starting with 93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351 not found: ID does not exist" containerID="93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"
Sep 17 19:23:09.439540 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.439506    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"} err="failed to get container status \"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351\": rpc error: code = NotFound desc = could not find container \"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351\": container with ID starting with 93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351 not found: ID does not exist"
Sep 17 19:23:09.439540 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.439533    2583 scope.go:117] "RemoveContainer" containerID="a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"
Sep 17 19:23:09.439810 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:23:09.439793    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc\": container with ID starting with a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc not found: ID does not exist" containerID="a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"
Sep 17 19:23:09.439870 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.439814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"} err="failed to get container status \"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc\": rpc error: code = NotFound desc = could not find container \"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc\": container with ID starting with a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc not found: ID does not exist"
Sep 17 19:23:09.439870 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.439828    2583 scope.go:117] "RemoveContainer" containerID="93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"
Sep 17 19:23:09.440039 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.440019    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351"} err="failed to get container status \"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351\": rpc error: code = NotFound desc = could not find container \"93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351\": container with ID starting with 93dc56c08c370bddb149ec585b64edc16f5756c7cfe9bbe2d151637ffc502351 not found: ID does not exist"
Sep 17 19:23:09.440085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.440042    2583 scope.go:117] "RemoveContainer" containerID="a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"
Sep 17 19:23:09.440257 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.440238    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc"} err="failed to get container status \"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc\": rpc error: code = NotFound desc = could not find container \"a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc\": container with ID starting with a35fc03b805462f8942f97a5fa8d342e7bcfc1be11f23e687ec2509936db8fdc not found: ID does not exist"
Sep 17 19:23:09.445503 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.445479    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"]
Sep 17 19:23:09.449443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.449423    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-g7knb"]
Sep 17 19:23:09.872188 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:09.872152    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"]
Sep 17 19:23:10.255779 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:10.255678    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" path="/var/lib/kubelet/pods/62c130a3-0d8d-472b-8390-c063d83b367b/volumes"
Sep 17 19:23:10.436417 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:23:10.436378    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:23:10.436417 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:10.436417 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:10.436417 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae" cmd=["/bin/sleep","15"]
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:10.436427    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" message="" err=<
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:23:10.436444    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" podUID="a865d6f9-3428-42cb-ab48-fb41e1d45a31" containerName="main" containerID="cri-o://92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae"
Sep 17 19:23:10.436712 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:10.436485    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" podUID="a865d6f9-3428-42cb-ab48-fb41e1d45a31" containerName="main" containerID="cri-o://92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae" gracePeriod=60
Sep 17 19:23:11.183329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.183306    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:11.276337 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276315    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tmp-dir\") pod \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") "
Sep 17 19:23:11.276454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276358    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-home\") pod \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") "
Sep 17 19:23:11.276454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276393    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tls-certs\") pod \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") "
Sep 17 19:23:11.276640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276457    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4htn\" (UniqueName: \"kubernetes.io/projected/a865d6f9-3428-42cb-ab48-fb41e1d45a31-kube-api-access-l4htn\") pod \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") "
Sep 17 19:23:11.276640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276487    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-model-cache\") pod \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") "
Sep 17 19:23:11.276640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276504    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-dshm\") pod \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\" (UID: \"a865d6f9-3428-42cb-ab48-fb41e1d45a31\") "
Sep 17 19:23:11.276640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276560    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a865d6f9-3428-42cb-ab48-fb41e1d45a31" (UID: "a865d6f9-3428-42cb-ab48-fb41e1d45a31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:11.276640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276577    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-home" (OuterVolumeSpecName: "home") pod "a865d6f9-3428-42cb-ab48-fb41e1d45a31" (UID: "a865d6f9-3428-42cb-ab48-fb41e1d45a31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:11.276901 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276759    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:11.276901 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276774    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:11.276901 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.276773    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-model-cache" (OuterVolumeSpecName: "model-cache") pod "a865d6f9-3428-42cb-ab48-fb41e1d45a31" (UID: "a865d6f9-3428-42cb-ab48-fb41e1d45a31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:11.278759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.278737    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-dshm" (OuterVolumeSpecName: "dshm") pod "a865d6f9-3428-42cb-ab48-fb41e1d45a31" (UID: "a865d6f9-3428-42cb-ab48-fb41e1d45a31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:11.278846 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.278784    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a865d6f9-3428-42cb-ab48-fb41e1d45a31" (UID: "a865d6f9-3428-42cb-ab48-fb41e1d45a31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:11.278889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.278869    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a865d6f9-3428-42cb-ab48-fb41e1d45a31-kube-api-access-l4htn" (OuterVolumeSpecName: "kube-api-access-l4htn") pod "a865d6f9-3428-42cb-ab48-fb41e1d45a31" (UID: "a865d6f9-3428-42cb-ab48-fb41e1d45a31"). InnerVolumeSpecName "kube-api-access-l4htn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:11.377980 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.377902    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4htn\" (UniqueName: \"kubernetes.io/projected/a865d6f9-3428-42cb-ab48-fb41e1d45a31-kube-api-access-l4htn\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:11.377980 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.377959    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:11.377980 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.377978    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a865d6f9-3428-42cb-ab48-fb41e1d45a31-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:11.378133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.377990    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a865d6f9-3428-42cb-ab48-fb41e1d45a31-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:23:11.433644 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.433619    2583 generic.go:356] "Generic (PLEG): container finished" podID="a865d6f9-3428-42cb-ab48-fb41e1d45a31" containerID="92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae" exitCode=0
Sep 17 19:23:11.433765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.433659    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"
Sep 17 19:23:11.433765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.433700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" event={"ID":"a865d6f9-3428-42cb-ab48-fb41e1d45a31","Type":"ContainerDied","Data":"92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae"}
Sep 17 19:23:11.433765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.433742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8" event={"ID":"a865d6f9-3428-42cb-ab48-fb41e1d45a31","Type":"ContainerDied","Data":"7d910eb03e241ff4cb1df839077365fd0e639e2d417b13338c9c96bf4f1296c0"}
Sep 17 19:23:11.433889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.433769    2583 scope.go:117] "RemoveContainer" containerID="92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae"
Sep 17 19:23:11.443724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.443515    2583 scope.go:117] "RemoveContainer" containerID="92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae"
Sep 17 19:23:11.443818 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:23:11.443788    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae\": container with ID starting with 92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae not found: ID does not exist" containerID="92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae"
Sep 17 19:23:11.443862 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.443811    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae"} err="failed to get container status \"92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae\": rpc error: code = NotFound desc = could not find container \"92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae\": container with ID starting with 92e8d382467f47cfc0d4835caf7bdb2ddf8817ba2566aeba303407a36386d3ae not found: ID does not exist"
Sep 17 19:23:11.455588 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.455548    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"]
Sep 17 19:23:11.461805 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:11.461782    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfxngt8"]
Sep 17 19:23:12.255651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:12.255615    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a865d6f9-3428-42cb-ab48-fb41e1d45a31" path="/var/lib/kubelet/pods/a865d6f9-3428-42cb-ab48-fb41e1d45a31/volumes"
Sep 17 19:23:15.968209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968175    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"]
Sep 17 19:23:15.968786 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968766    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="main"
Sep 17 19:23:15.968882 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968788    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="main"
Sep 17 19:23:15.968882 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968803    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a865d6f9-3428-42cb-ab48-fb41e1d45a31" containerName="main"
Sep 17 19:23:15.968882 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968811    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a865d6f9-3428-42cb-ab48-fb41e1d45a31" containerName="main"
Sep 17 19:23:15.968882 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968850    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="llm-d-routing-sidecar"
Sep 17 19:23:15.968882 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968859    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="llm-d-routing-sidecar"
Sep 17 19:23:15.969146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968945    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a865d6f9-3428-42cb-ab48-fb41e1d45a31" containerName="main"
Sep 17 19:23:15.969146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968960    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="llm-d-routing-sidecar"
Sep 17 19:23:15.969146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.968970    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c130a3-0d8d-472b-8390-c063d83b367b" containerName="main"
Sep 17 19:23:15.973577 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.973555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:15.976323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.976307    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 19:23:15.976616 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.976592    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 19:23:15.976800 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.976625    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 19:23:15.977979 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.977953    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-l82t5\""
Sep 17 19:23:15.978098 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.977996    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 19:23:15.978442 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.978423    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:23:15.989090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:15.989068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"]
Sep 17 19:23:16.111663 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.111634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mktv4\" (UniqueName: \"kubernetes.io/projected/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kube-api-access-mktv4\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.111811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.111679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-model-cache\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.111811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.111747    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kserve-provision-location\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.111900 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.111825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-home\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.111900 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.111862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tmp-dir\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.111900 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.111887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-cabundle-cert\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.112046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.111908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-dshm\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.112046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.112010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tls-certs\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213227 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tmp-dir\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213227 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-cabundle-cert\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213398 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-dshm\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213398 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tls-certs\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mktv4\" (UniqueName: \"kubernetes.io/projected/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kube-api-access-mktv4\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213517 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-model-cache\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213554 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kserve-provision-location\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213611 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-home\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213669 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tmp-dir\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.213943 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-home\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.214252 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.213916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kserve-provision-location\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.214252 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.214084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-cabundle-cert\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.214252 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.214109    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-model-cache\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.215658 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.215636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-dshm\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.215943 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.215924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tls-certs\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.221498 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.221447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mktv4\" (UniqueName: \"kubernetes.io/projected/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kube-api-access-mktv4\") pod \"auth-enabled-test-kserve-798758fc4d-5g8nt\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.289837 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.289814    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:16.577113 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:23:16.577077    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb196ee35_9a45_4ae1_9686_e647bfcdc5f6.slice/crio-b568e4eb9c62707854179ff1ed2b3657284cfb4bcde4eba53086b8e290dedb81 WatchSource:0}: Error finding container b568e4eb9c62707854179ff1ed2b3657284cfb4bcde4eba53086b8e290dedb81: Status 404 returned error can't find the container with id b568e4eb9c62707854179ff1ed2b3657284cfb4bcde4eba53086b8e290dedb81
Sep 17 19:23:16.577266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:16.577190    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"]
Sep 17 19:23:17.471553 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:17.471514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" event={"ID":"b196ee35-9a45-4ae1-9686-e647bfcdc5f6","Type":"ContainerStarted","Data":"df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608"}
Sep 17 19:23:17.471930 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:17.471575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" event={"ID":"b196ee35-9a45-4ae1-9686-e647bfcdc5f6","Type":"ContainerStarted","Data":"b568e4eb9c62707854179ff1ed2b3657284cfb4bcde4eba53086b8e290dedb81"}
Sep 17 19:23:19.484231 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:19.484200    2583 generic.go:356] "Generic (PLEG): container finished" podID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerID="df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608" exitCode=0
Sep 17 19:23:19.484595 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:19.484261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" event={"ID":"b196ee35-9a45-4ae1-9686-e647bfcdc5f6","Type":"ContainerDied","Data":"df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608"}
Sep 17 19:23:47.692464 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:47.692422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" event={"ID":"b196ee35-9a45-4ae1-9686-e647bfcdc5f6","Type":"ContainerStarted","Data":"23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2"}
Sep 17 19:23:47.716242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:47.716194    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podStartSLOduration=3.25838347 podStartE2EDuration="32.716178238s" totalImagesPullingTime="27.216217759s" totalInitContainerRuntime="2.241577009s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:47.71416948 +0000 UTC m=+1029.762049333" watchObservedRunningTime="2026-09-17 19:23:47.716178238 +0000 UTC m=+1029.764058096"
Sep 17 19:23:48.290606 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:48.290563    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:56.290880 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:56.290844    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:23:56.292310 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:23:56.292281    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:24:04.015270 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.015229    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"]
Sep 17 19:24:04.293158 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.293073    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"]
Sep 17 19:24:04.293315 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.293185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.297406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.297106    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 19:24:04.297406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.297157    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 19:24:04.297634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.297600    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-hptdm\""
Sep 17 19:24:04.297793 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.297738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 19:24:04.467680 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.467637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.467875 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.467725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.467875 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.467761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.467875 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.467845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.467875 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.467868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzzk\" (UniqueName: \"kubernetes.io/projected/6f100eba-293b-4211-8b3c-8ef163c27eac-kube-api-access-fpzzk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.468033 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.467894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f100eba-293b-4211-8b3c-8ef163c27eac-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f100eba-293b-4211-8b3c-8ef163c27eac-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569188    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzzk\" (UniqueName: \"kubernetes.io/projected/6f100eba-293b-4211-8b3c-8ef163c27eac-kube-api-access-fpzzk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569646    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.569811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.569721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.571753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.571729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.571949 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.571934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f100eba-293b-4211-8b3c-8ef163c27eac-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.578452 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.578421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzzk\" (UniqueName: \"kubernetes.io/projected/6f100eba-293b-4211-8b3c-8ef163c27eac-kube-api-access-fpzzk\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.605395 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.605368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:04.908725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:04.908679    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"]
Sep 17 19:24:04.911223 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:24:04.911181    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f100eba_293b_4211_8b3c_8ef163c27eac.slice/crio-ba2849a8f2fbda873be28b42ba90586cf28406890bb87050b9f2cd7479f3ed11 WatchSource:0}: Error finding container ba2849a8f2fbda873be28b42ba90586cf28406890bb87050b9f2cd7479f3ed11: Status 404 returned error can't find the container with id ba2849a8f2fbda873be28b42ba90586cf28406890bb87050b9f2cd7479f3ed11
Sep 17 19:24:05.807789 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:05.807750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" event={"ID":"6f100eba-293b-4211-8b3c-8ef163c27eac","Type":"ContainerStarted","Data":"a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2"}
Sep 17 19:24:05.808237 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:05.807816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" event={"ID":"6f100eba-293b-4211-8b3c-8ef163c27eac","Type":"ContainerStarted","Data":"ba2849a8f2fbda873be28b42ba90586cf28406890bb87050b9f2cd7479f3ed11"}
Sep 17 19:24:05.826729 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:05.826665    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" podStartSLOduration=2.826644794 podStartE2EDuration="2.826644794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:05.824847172 +0000 UTC m=+1047.872727030" watchObservedRunningTime="2026-09-17 19:24:05.826644794 +0000 UTC m=+1047.874524652"
Sep 17 19:24:06.290808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:06.290728    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:24:06.605465 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:06.605432    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:14.606246 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:14.606212    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:14.618677 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:14.618647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:14.870442 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:14.870361    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:16.291018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:16.290981    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:24:26.290791 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:26.290681    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:24:36.290875 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:36.290834    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:24:39.040871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.040833    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"]
Sep 17 19:24:39.051124 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:24:39.051089    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:24:39.051124 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:24:39.051124 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:24:39.051124 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2" cmd=["/bin/sleep","15"]
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.051149    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" message="" err=<
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:24:39.051167    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" podUID="6f100eba-293b-4211-8b3c-8ef163c27eac" containerName="main" containerID="cri-o://a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2"
Sep 17 19:24:39.051415 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.051212    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" podUID="6f100eba-293b-4211-8b3c-8ef163c27eac" containerName="main" containerID="cri-o://a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2" gracePeriod=60
Sep 17 19:24:39.417197 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.417171    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:39.458810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.458782    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpzzk\" (UniqueName: \"kubernetes.io/projected/6f100eba-293b-4211-8b3c-8ef163c27eac-kube-api-access-fpzzk\") pod \"6f100eba-293b-4211-8b3c-8ef163c27eac\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") "
Sep 17 19:24:39.458968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.458845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-tmp-dir\") pod \"6f100eba-293b-4211-8b3c-8ef163c27eac\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") "
Sep 17 19:24:39.458968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.458896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-model-cache\") pod \"6f100eba-293b-4211-8b3c-8ef163c27eac\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") "
Sep 17 19:24:39.458968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.458938    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-dshm\") pod \"6f100eba-293b-4211-8b3c-8ef163c27eac\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") "
Sep 17 19:24:39.458968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.458959    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-home\") pod \"6f100eba-293b-4211-8b3c-8ef163c27eac\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") "
Sep 17 19:24:39.459188 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.458985    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f100eba-293b-4211-8b3c-8ef163c27eac-tls-certs\") pod \"6f100eba-293b-4211-8b3c-8ef163c27eac\" (UID: \"6f100eba-293b-4211-8b3c-8ef163c27eac\") "
Sep 17 19:24:39.459188 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.459151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f100eba-293b-4211-8b3c-8ef163c27eac" (UID: "6f100eba-293b-4211-8b3c-8ef163c27eac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.459299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.459183    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f100eba-293b-4211-8b3c-8ef163c27eac" (UID: "6f100eba-293b-4211-8b3c-8ef163c27eac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.459352 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.459328    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.459411 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.459351    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.459411 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.459396    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-home" (OuterVolumeSpecName: "home") pod "6f100eba-293b-4211-8b3c-8ef163c27eac" (UID: "6f100eba-293b-4211-8b3c-8ef163c27eac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.461288 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.461257    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f100eba-293b-4211-8b3c-8ef163c27eac-kube-api-access-fpzzk" (OuterVolumeSpecName: "kube-api-access-fpzzk") pod "6f100eba-293b-4211-8b3c-8ef163c27eac" (UID: "6f100eba-293b-4211-8b3c-8ef163c27eac"). InnerVolumeSpecName "kube-api-access-fpzzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:24:39.461288 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.461274    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f100eba-293b-4211-8b3c-8ef163c27eac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f100eba-293b-4211-8b3c-8ef163c27eac" (UID: "6f100eba-293b-4211-8b3c-8ef163c27eac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:24:39.461431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.461376    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-dshm" (OuterVolumeSpecName: "dshm") pod "6f100eba-293b-4211-8b3c-8ef163c27eac" (UID: "6f100eba-293b-4211-8b3c-8ef163c27eac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:39.559993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.559961    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.559993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.559987    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f100eba-293b-4211-8b3c-8ef163c27eac-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.559993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.559997    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f100eba-293b-4211-8b3c-8ef163c27eac-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:24:39.560218 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:39.560006    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpzzk\" (UniqueName: \"kubernetes.io/projected/6f100eba-293b-4211-8b3c-8ef163c27eac-kube-api-access-fpzzk\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:24:40.017293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.017254    2583 generic.go:356] "Generic (PLEG): container finished" podID="6f100eba-293b-4211-8b3c-8ef163c27eac" containerID="a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2" exitCode=0
Sep 17 19:24:40.017484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.017311    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"
Sep 17 19:24:40.017484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.017375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" event={"ID":"6f100eba-293b-4211-8b3c-8ef163c27eac","Type":"ContainerDied","Data":"a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2"}
Sep 17 19:24:40.017484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.017424    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc" event={"ID":"6f100eba-293b-4211-8b3c-8ef163c27eac","Type":"ContainerDied","Data":"ba2849a8f2fbda873be28b42ba90586cf28406890bb87050b9f2cd7479f3ed11"}
Sep 17 19:24:40.017484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.017451    2583 scope.go:117] "RemoveContainer" containerID="a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2"
Sep 17 19:24:40.028669 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.028649    2583 scope.go:117] "RemoveContainer" containerID="a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2"
Sep 17 19:24:40.028989 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:24:40.028968    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2\": container with ID starting with a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2 not found: ID does not exist" containerID="a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2"
Sep 17 19:24:40.029064 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.029001    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2"} err="failed to get container status \"a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2\": rpc error: code = NotFound desc = could not find container \"a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2\": container with ID starting with a562f5164b263b8c72a8ddfa5330dcbfedc458cfd2952ae53c7130be74930ee2 not found: ID does not exist"
Sep 17 19:24:40.043165 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.043142    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"]
Sep 17 19:24:40.050038 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.050016    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64446wghc"]
Sep 17 19:24:40.257663 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:40.257631    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f100eba-293b-4211-8b3c-8ef163c27eac" path="/var/lib/kubelet/pods/6f100eba-293b-4211-8b3c-8ef163c27eac/volumes"
Sep 17 19:24:46.290500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:46.290451    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:24:56.291129 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:24:56.291089    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:25:06.291204 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:06.291164    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:25:16.290241 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:16.290199    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:25:26.291093 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:26.291041    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:25:36.300215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:36.300181    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:25:36.308171 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:36.308145    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:25:37.273975 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.273940    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"]
Sep 17 19:25:37.274335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.274322    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f100eba-293b-4211-8b3c-8ef163c27eac" containerName="main"
Sep 17 19:25:37.274384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.274336    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f100eba-293b-4211-8b3c-8ef163c27eac" containerName="main"
Sep 17 19:25:37.274424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.274404    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f100eba-293b-4211-8b3c-8ef163c27eac" containerName="main"
Sep 17 19:25:37.278847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.278827    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.282255 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.282233    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-wrwx8\""
Sep 17 19:25:37.282370 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.282310    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 19:25:37.282370 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.282310    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 19:25:37.282546 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.282315    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 19:25:37.282546 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.282315    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 19:25:37.282546 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.282499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:25:37.289192 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.289174    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"]
Sep 17 19:25:37.364322 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-home\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.364322 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tls-certs\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.364808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.364808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.364808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6587d98c-d853-4dfc-9f17-43c12bb6cc18-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.364808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-model-cache\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.364808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtkb\" (UniqueName: \"kubernetes.io/projected/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kube-api-access-rgtkb\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.364808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.364629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-dshm\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.465630 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-home\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.465805 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tls-certs\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.465805 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.465805 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.465805 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6587d98c-d853-4dfc-9f17-43c12bb6cc18-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-model-cache\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtkb\" (UniqueName: \"kubernetes.io/projected/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kube-api-access-rgtkb\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.465926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-dshm\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.466050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-home\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.466077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.466146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.466226    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-model-cache\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.466663 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.466622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6587d98c-d853-4dfc-9f17-43c12bb6cc18-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.468151 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.468133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-dshm\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.468453 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.468431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tls-certs\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.474123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.474093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtkb\" (UniqueName: \"kubernetes.io/projected/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kube-api-access-rgtkb\") pod \"tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.524254 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.523882    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"]
Sep 17 19:25:37.529625 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.529606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.537993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.537969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"]
Sep 17 19:25:37.590335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.590308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:37.668333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.668303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.668446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.668402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0551e4d-5042-4b57-8795-4f0c3233e5fe-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.668446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.668436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.668523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.668503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.769428 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.769400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.769559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.769443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0551e4d-5042-4b57-8795-4f0c3233e5fe-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.769559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.769463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.769559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.769497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.769918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.769895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.770056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.769937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.770162 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.770137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0551e4d-5042-4b57-8795-4f0c3233e5fe-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.772115 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.772097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.842294 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.842276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:25:37.867387 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:37.867362    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"]
Sep 17 19:25:37.868450 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:25:37.868353    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6587d98c_d853_4dfc_9f17_43c12bb6cc18.slice/crio-4240360538fb77968c3e24daf7dc9128a77485e17093889f44654cf3eccde09e WatchSource:0}: Error finding container 4240360538fb77968c3e24daf7dc9128a77485e17093889f44654cf3eccde09e: Status 404 returned error can't find the container with id 4240360538fb77968c3e24daf7dc9128a77485e17093889f44654cf3eccde09e
Sep 17 19:25:38.122628 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:38.122603    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"]
Sep 17 19:25:38.125394 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:25:38.125363    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0551e4d_5042_4b57_8795_4f0c3233e5fe.slice/crio-05451ba85e998d09fd5c02c0614c1162c05c3b5615b6cf94e5019bf048251bc9 WatchSource:0}: Error finding container 05451ba85e998d09fd5c02c0614c1162c05c3b5615b6cf94e5019bf048251bc9: Status 404 returned error can't find the container with id 05451ba85e998d09fd5c02c0614c1162c05c3b5615b6cf94e5019bf048251bc9
Sep 17 19:25:38.366993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:38.366952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" event={"ID":"c0551e4d-5042-4b57-8795-4f0c3233e5fe","Type":"ContainerStarted","Data":"da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715"}
Sep 17 19:25:38.367470 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:38.367012    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" event={"ID":"c0551e4d-5042-4b57-8795-4f0c3233e5fe","Type":"ContainerStarted","Data":"05451ba85e998d09fd5c02c0614c1162c05c3b5615b6cf94e5019bf048251bc9"}
Sep 17 19:25:38.368825 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:38.368793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" event={"ID":"6587d98c-d853-4dfc-9f17-43c12bb6cc18","Type":"ContainerStarted","Data":"81042fbef350f9ddf661516d0beed5ace26f1e59a74f41b926238b9a525e5cc1"}
Sep 17 19:25:38.368963 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:38.368857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" event={"ID":"6587d98c-d853-4dfc-9f17-43c12bb6cc18","Type":"ContainerStarted","Data":"4240360538fb77968c3e24daf7dc9128a77485e17093889f44654cf3eccde09e"}
Sep 17 19:25:41.386642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:41.386603    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerID="da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715" exitCode=0
Sep 17 19:25:41.387054 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:41.386676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" event={"ID":"c0551e4d-5042-4b57-8795-4f0c3233e5fe","Type":"ContainerDied","Data":"da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715"}
Sep 17 19:25:41.387972 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:41.387948    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:25:41.388503 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:41.388403    2583 generic.go:356] "Generic (PLEG): container finished" podID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerID="81042fbef350f9ddf661516d0beed5ace26f1e59a74f41b926238b9a525e5cc1" exitCode=0
Sep 17 19:25:41.388503 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:41.388459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" event={"ID":"6587d98c-d853-4dfc-9f17-43c12bb6cc18","Type":"ContainerDied","Data":"81042fbef350f9ddf661516d0beed5ace26f1e59a74f41b926238b9a525e5cc1"}
Sep 17 19:25:42.395653 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:42.395616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" event={"ID":"6587d98c-d853-4dfc-9f17-43c12bb6cc18","Type":"ContainerStarted","Data":"31d88d973de0e92efaf62c0c6bcc8b03a3eedfcb5764abfe78042542547e29ec"}
Sep 17 19:25:42.418240 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:42.418192    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" podStartSLOduration=2.8784824 podStartE2EDuration="5.418177184s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.539694784s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:25:42.417438618 +0000 UTC m=+1144.465318474" watchObservedRunningTime="2026-09-17 19:25:42.418177184 +0000 UTC m=+1144.466057040"
Sep 17 19:25:42.591113 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:42.591080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:47.591024 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:47.590984    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:47.610138 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:47.610106    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:25:48.459339 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:25:48.459257    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:26:31.756101 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:26:31.756065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" event={"ID":"c0551e4d-5042-4b57-8795-4f0c3233e5fe","Type":"ContainerStarted","Data":"bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855"}
Sep 17 19:26:31.756785 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:26:31.756300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:26:31.777235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:26:31.777187    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" podStartSLOduration=2.661224886 podStartE2EDuration="54.777172797s" totalImagesPullingTime="49.590779597s" totalInitContainerRuntime="2.525168314s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:26:31.774807397 +0000 UTC m=+1193.822687257" watchObservedRunningTime="2026-09-17 19:26:31.777172797 +0000 UTC m=+1193.825052717"
Sep 17 19:27:02.773427 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:02.773393    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:27:06.623992 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:06.623958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"]
Sep 17 19:27:07.599380 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:07.599348    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv_c0551e4d-5042-4b57-8795-4f0c3233e5fe/main/0.log"
Sep 17 19:27:08.641212 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:08.641171    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"]
Sep 17 19:27:08.641666 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:08.641460    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" podUID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerName="main" containerID="cri-o://bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855" gracePeriod=30
Sep 17 19:27:08.648741 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:08.648712    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"]
Sep 17 19:27:08.657877 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:27:08.657842    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:27:08.657877 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:27:08.657877 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:27:08.657877 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="31d88d973de0e92efaf62c0c6bcc8b03a3eedfcb5764abfe78042542547e29ec" cmd=["/bin/sleep","15"]
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:08.657897    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" message="" err=<
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:27:08.657914    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerName="main" containerID="cri-o://31d88d973de0e92efaf62c0c6bcc8b03a3eedfcb5764abfe78042542547e29ec"
Sep 17 19:27:08.658105 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:08.657965    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerName="main" containerID="cri-o://31d88d973de0e92efaf62c0c6bcc8b03a3eedfcb5764abfe78042542547e29ec" gracePeriod=60
Sep 17 19:27:08.972642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:08.972609    2583 generic.go:356] "Generic (PLEG): container finished" podID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerID="31d88d973de0e92efaf62c0c6bcc8b03a3eedfcb5764abfe78042542547e29ec" exitCode=0
Sep 17 19:27:08.972823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:08.972670    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" event={"ID":"6587d98c-d853-4dfc-9f17-43c12bb6cc18","Type":"ContainerDied","Data":"31d88d973de0e92efaf62c0c6bcc8b03a3eedfcb5764abfe78042542547e29ec"}
Sep 17 19:27:09.022266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.022239    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:27:09.117934 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.117908    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-home\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.117950    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgtkb\" (UniqueName: \"kubernetes.io/projected/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kube-api-access-rgtkb\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.117992    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tls-certs\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118128 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-model-cache\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118128 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118069    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6587d98c-d853-4dfc-9f17-43c12bb6cc18-cabundle-cert\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-dshm\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118208    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-home" (OuterVolumeSpecName: "home") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.118397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118296    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-model-cache" (OuterVolumeSpecName: "model-cache") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.118397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118321    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kserve-provision-location\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118375    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6587d98c-d853-4dfc-9f17-43c12bb6cc18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:09.118561 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118395    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tmp-dir\") pod \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\" (UID: \"6587d98c-d853-4dfc-9f17-43c12bb6cc18\") "
Sep 17 19:27:09.118630 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118608    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.118786 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118765    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.118857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118808    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.118973 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118915    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.119038 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.118980    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6587d98c-d853-4dfc-9f17-43c12bb6cc18-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.131205 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.131166    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:09.139385 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.139358    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.151081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.151048    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-dshm" (OuterVolumeSpecName: "dshm") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.151263 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.151247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kube-api-access-rgtkb" (OuterVolumeSpecName: "kube-api-access-rgtkb") pod "6587d98c-d853-4dfc-9f17-43c12bb6cc18" (UID: "6587d98c-d853-4dfc-9f17-43c12bb6cc18"). InnerVolumeSpecName "kube-api-access-rgtkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:09.220468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.220436    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.220605 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.220471    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtkb\" (UniqueName: \"kubernetes.io/projected/6587d98c-d853-4dfc-9f17-43c12bb6cc18-kube-api-access-rgtkb\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.220605 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.220486    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6587d98c-d853-4dfc-9f17-43c12bb6cc18-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.220605 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.220498    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6587d98c-d853-4dfc-9f17-43c12bb6cc18-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.286178 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.286154    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:27:09.422276 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.422243    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-kserve-provision-location\") pod \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") "
Sep 17 19:27:09.422455 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.422335    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tmp-dir\") pod \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") "
Sep 17 19:27:09.422455 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.422414    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tls-certs\") pod \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") "
Sep 17 19:27:09.422455 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.422446    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0551e4d-5042-4b57-8795-4f0c3233e5fe-cabundle-cert\") pod \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\" (UID: \"c0551e4d-5042-4b57-8795-4f0c3233e5fe\") "
Sep 17 19:27:09.422811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.422777    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0551e4d-5042-4b57-8795-4f0c3233e5fe" (UID: "c0551e4d-5042-4b57-8795-4f0c3233e5fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.422930 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.422845    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0551e4d-5042-4b57-8795-4f0c3233e5fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0551e4d-5042-4b57-8795-4f0c3233e5fe" (UID: "c0551e4d-5042-4b57-8795-4f0c3233e5fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:09.424538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.424522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0551e4d-5042-4b57-8795-4f0c3233e5fe" (UID: "c0551e4d-5042-4b57-8795-4f0c3233e5fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:09.443252 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.443230    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0551e4d-5042-4b57-8795-4f0c3233e5fe" (UID: "c0551e4d-5042-4b57-8795-4f0c3233e5fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:09.524088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.524023    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.524088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.524054    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0551e4d-5042-4b57-8795-4f0c3233e5fe-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.524088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.524065    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.524088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.524074    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0551e4d-5042-4b57-8795-4f0c3233e5fe-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:09.979708 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.979661    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"
Sep 17 19:27:09.980109 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.979712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m" event={"ID":"6587d98c-d853-4dfc-9f17-43c12bb6cc18","Type":"ContainerDied","Data":"4240360538fb77968c3e24daf7dc9128a77485e17093889f44654cf3eccde09e"}
Sep 17 19:27:09.980109 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.979771    2583 scope.go:117] "RemoveContainer" containerID="31d88d973de0e92efaf62c0c6bcc8b03a3eedfcb5764abfe78042542547e29ec"
Sep 17 19:27:09.981716 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.981666    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerID="bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855" exitCode=0
Sep 17 19:27:09.981716 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.981724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" event={"ID":"c0551e4d-5042-4b57-8795-4f0c3233e5fe","Type":"ContainerDied","Data":"bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855"}
Sep 17 19:27:09.981913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.981739    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"
Sep 17 19:27:09.981913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.981763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv" event={"ID":"c0551e4d-5042-4b57-8795-4f0c3233e5fe","Type":"ContainerDied","Data":"05451ba85e998d09fd5c02c0614c1162c05c3b5615b6cf94e5019bf048251bc9"}
Sep 17 19:27:09.989957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.989937    2583 scope.go:117] "RemoveContainer" containerID="81042fbef350f9ddf661516d0beed5ace26f1e59a74f41b926238b9a525e5cc1"
Sep 17 19:27:09.998058 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:09.998040    2583 scope.go:117] "RemoveContainer" containerID="bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855"
Sep 17 19:27:10.008974 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.008950    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"]
Sep 17 19:27:10.009083 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.008981    2583 scope.go:117] "RemoveContainer" containerID="da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715"
Sep 17 19:27:10.013426 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.013404    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-8f7f79dcb-xck7m"]
Sep 17 19:27:10.018122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.018104    2583 scope.go:117] "RemoveContainer" containerID="bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855"
Sep 17 19:27:10.018400 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:27:10.018382    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855\": container with ID starting with bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855 not found: ID does not exist" containerID="bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855"
Sep 17 19:27:10.018460 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.018409    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855"} err="failed to get container status \"bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855\": rpc error: code = NotFound desc = could not find container \"bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855\": container with ID starting with bf59fe2ceb72f996bc9a7aff4e7f9b4100747bd6f3be62a999f6263ab48ab855 not found: ID does not exist"
Sep 17 19:27:10.018460 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.018432    2583 scope.go:117] "RemoveContainer" containerID="da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715"
Sep 17 19:27:10.018651 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:27:10.018637    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715\": container with ID starting with da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715 not found: ID does not exist" containerID="da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715"
Sep 17 19:27:10.018712 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.018657    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715"} err="failed to get container status \"da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715\": rpc error: code = NotFound desc = could not find container \"da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715\": container with ID starting with da36aaf8c32cac07e77af05dcc0a660d8741b2b4e8182b9d59608a65432c5715 not found: ID does not exist"
Sep 17 19:27:10.023484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.023461    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"]
Sep 17 19:27:10.027085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.027068    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-677f47d5f5-mvpnv"]
Sep 17 19:27:10.255262 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.255195    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" path="/var/lib/kubelet/pods/6587d98c-d853-4dfc-9f17-43c12bb6cc18/volumes"
Sep 17 19:27:10.255609 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:10.255595    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" path="/var/lib/kubelet/pods/c0551e4d-5042-4b57-8795-4f0c3233e5fe/volumes"
Sep 17 19:27:21.652159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:21.652037    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" containerID="cri-o://23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2" gracePeriod=45
Sep 17 19:27:22.301020 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.300983    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 19:27:22.421574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.421519    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:27:22.527416 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kserve-provision-location\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.527555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527424    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-home\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.527555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mktv4\" (UniqueName: \"kubernetes.io/projected/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kube-api-access-mktv4\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.527555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-cabundle-cert\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.527555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527516    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tmp-dir\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.527555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527542    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-dshm\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.527838 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527583    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tls-certs\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.527838 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.527655    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-model-cache\") pod \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\" (UID: \"b196ee35-9a45-4ae1-9686-e647bfcdc5f6\") "
Sep 17 19:27:22.528118 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.528096    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:22.528232 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.528001    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:22.528333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.528213    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-home" (OuterVolumeSpecName: "home") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:22.529935 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.529913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kube-api-access-mktv4" (OuterVolumeSpecName: "kube-api-access-mktv4") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "kube-api-access-mktv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:22.530533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.530514    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-dshm" (OuterVolumeSpecName: "dshm") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:22.530614 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.530516    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:22.546388 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.546363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:22.554419 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.554399    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b196ee35-9a45-4ae1-9686-e647bfcdc5f6" (UID: "b196ee35-9a45-4ae1-9686-e647bfcdc5f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:22.628471 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628439    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:22.628471 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628473    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:22.628471 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628484    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:22.628634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628491    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:22.628634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628500    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:22.628634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628508    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:22.628634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628518    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mktv4\" (UniqueName: \"kubernetes.io/projected/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-kube-api-access-mktv4\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:22.628634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:22.628525    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b196ee35-9a45-4ae1-9686-e647bfcdc5f6-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:27:23.062350 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.062312    2583 generic.go:356] "Generic (PLEG): container finished" podID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerID="23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2" exitCode=0
Sep 17 19:27:23.062836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.062375    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"
Sep 17 19:27:23.062836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.062399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" event={"ID":"b196ee35-9a45-4ae1-9686-e647bfcdc5f6","Type":"ContainerDied","Data":"23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2"}
Sep 17 19:27:23.062836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.062458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt" event={"ID":"b196ee35-9a45-4ae1-9686-e647bfcdc5f6","Type":"ContainerDied","Data":"b568e4eb9c62707854179ff1ed2b3657284cfb4bcde4eba53086b8e290dedb81"}
Sep 17 19:27:23.062836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.062482    2583 scope.go:117] "RemoveContainer" containerID="23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2"
Sep 17 19:27:23.072733 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.072713    2583 scope.go:117] "RemoveContainer" containerID="df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608"
Sep 17 19:27:23.080860 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.080842    2583 scope.go:117] "RemoveContainer" containerID="23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2"
Sep 17 19:27:23.081134 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:27:23.081116    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2\": container with ID starting with 23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2 not found: ID does not exist" containerID="23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2"
Sep 17 19:27:23.081204 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.081142    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2"} err="failed to get container status \"23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2\": rpc error: code = NotFound desc = could not find container \"23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2\": container with ID starting with 23f8c7b94154483d1a39e2d166a1d03704f6b5b5461c06bdcf2c91598521b6d2 not found: ID does not exist"
Sep 17 19:27:23.081204 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.081161    2583 scope.go:117] "RemoveContainer" containerID="df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608"
Sep 17 19:27:23.081428 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:27:23.081412    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608\": container with ID starting with df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608 not found: ID does not exist" containerID="df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608"
Sep 17 19:27:23.081477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.081431    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608"} err="failed to get container status \"df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608\": rpc error: code = NotFound desc = could not find container \"df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608\": container with ID starting with df7d16f660c169b830db10bb407c78f6387ea4c5605b591171824ae14cf1c608 not found: ID does not exist"
Sep 17 19:27:23.085972 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.085952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"]
Sep 17 19:27:23.090546 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:23.090525    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-798758fc4d-5g8nt"]
Sep 17 19:27:24.255109 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:27:24.255081    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" path="/var/lib/kubelet/pods/b196ee35-9a45-4ae1-9686-e647bfcdc5f6/volumes"
Sep 17 19:28:10.166759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.166728    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"]
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167119    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerName="storage-initializer"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167131    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerName="storage-initializer"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167139    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167145    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167155    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167161    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167172    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167176    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167184    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="storage-initializer"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167190    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="storage-initializer"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167205    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerName="storage-initializer"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167210    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerName="storage-initializer"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167260    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b196ee35-9a45-4ae1-9686-e647bfcdc5f6" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167268    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6587d98c-d853-4dfc-9f17-43c12bb6cc18" containerName="main"
Sep 17 19:28:10.168112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.167274    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0551e4d-5042-4b57-8795-4f0c3233e5fe" containerName="main"
Sep 17 19:28:10.169154 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.169139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.172025 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.172001    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 19:28:10.172025 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.172023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 19:28:10.172241 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.172027    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-xrhg4\""
Sep 17 19:28:10.172241 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.172027    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 19:28:10.172358 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.172312    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:28:10.172417 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.172380    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 19:28:10.184874 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.184853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"]
Sep 17 19:28:10.294160 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-home\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.294160 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca3ab363-1673-415c-a400-02f52f14ef74-cabundle-cert\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.294377 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294181    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ab363-1673-415c-a400-02f52f14ef74-tls-certs\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.294377 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.294377 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchm7\" (UniqueName: \"kubernetes.io/projected/ca3ab363-1673-415c-a400-02f52f14ef74-kube-api-access-zchm7\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.294511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-tmp-dir\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.294511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-dshm\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.294511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.294441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-model-cache\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.395816 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.395788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-home\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.395966 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.395823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca3ab363-1673-415c-a400-02f52f14ef74-cabundle-cert\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.395966 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.395851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ab363-1673-415c-a400-02f52f14ef74-tls-certs\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.395966 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.395895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.395966 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.395953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchm7\" (UniqueName: \"kubernetes.io/projected/ca3ab363-1673-415c-a400-02f52f14ef74-kube-api-access-zchm7\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396199 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-tmp-dir\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396199 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-dshm\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396199 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-model-cache\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396362 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-home\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396585 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396560    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-model-cache\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-tmp-dir\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.396948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.396677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca3ab363-1673-415c-a400-02f52f14ef74-cabundle-cert\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.399110 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.399079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ab363-1673-415c-a400-02f52f14ef74-tls-certs\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.399110 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.399099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-dshm\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.409194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.409171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchm7\" (UniqueName: \"kubernetes.io/projected/ca3ab363-1673-415c-a400-02f52f14ef74-kube-api-access-zchm7\") pod \"auth-invalid-token-test-kserve-58d4c47bbd-9hlqs\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.480756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.480680    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:10.763949 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:10.763918    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"]
Sep 17 19:28:10.765572 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:28:10.765544    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3ab363_1673_415c_a400_02f52f14ef74.slice/crio-4ca210a7f75b2557c404e213266b1b3983f2b452f6a10c59b3af6cd2d43edebb WatchSource:0}: Error finding container 4ca210a7f75b2557c404e213266b1b3983f2b452f6a10c59b3af6cd2d43edebb: Status 404 returned error can't find the container with id 4ca210a7f75b2557c404e213266b1b3983f2b452f6a10c59b3af6cd2d43edebb
Sep 17 19:28:11.331082 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:11.331048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" event={"ID":"ca3ab363-1673-415c-a400-02f52f14ef74","Type":"ContainerStarted","Data":"b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c"}
Sep 17 19:28:11.331502 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:11.331103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" event={"ID":"ca3ab363-1673-415c-a400-02f52f14ef74","Type":"ContainerStarted","Data":"4ca210a7f75b2557c404e213266b1b3983f2b452f6a10c59b3af6cd2d43edebb"}
Sep 17 19:28:13.342808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:13.342780    2583 generic.go:356] "Generic (PLEG): container finished" podID="ca3ab363-1673-415c-a400-02f52f14ef74" containerID="b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c" exitCode=0
Sep 17 19:28:13.343145 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:13.342849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" event={"ID":"ca3ab363-1673-415c-a400-02f52f14ef74","Type":"ContainerDied","Data":"b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c"}
Sep 17 19:28:14.352138 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:14.352097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" event={"ID":"ca3ab363-1673-415c-a400-02f52f14ef74","Type":"ContainerStarted","Data":"dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc"}
Sep 17 19:28:14.377425 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:14.377365    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podStartSLOduration=2.027456082 podStartE2EDuration="4.377344936s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.349888854s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:14.372649931 +0000 UTC m=+1296.420529788" watchObservedRunningTime="2026-09-17 19:28:14.377344936 +0000 UTC m=+1296.425224796"
Sep 17 19:28:14.481212 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:14.481180    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:20.481471 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:20.481425    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:28:20.483100 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:20.483072    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:28:30.481569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:30.481526    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:28:40.482084 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:40.482039    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:28:43.986016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.985977    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"]
Sep 17 19:28:43.990644 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.990623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:43.993233 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.993210    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 19:28:43.993424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.993408    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 19:28:43.993505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.993437    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:28:43.993505 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.993408    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 19:28:43.994389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.994369    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9rs56\""
Sep 17 19:28:43.994512 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.994441    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 19:28:43.998798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:43.998776    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"]
Sep 17 19:28:44.089501 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.089632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.089632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-dshm\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.089632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a2902f-a1ff-4ac0-9079-809421c69a6a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.089632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjfw5\" (UniqueName: \"kubernetes.io/projected/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kube-api-access-gjfw5\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.089632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.089836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.089836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.089682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-home\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190480 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-home\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190753 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-dshm\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190809 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a2902f-a1ff-4ac0-9079-809421c69a6a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190809 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfw5\" (UniqueName: \"kubernetes.io/projected/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kube-api-access-gjfw5\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.190915 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.191023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.190988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-home\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.191088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.191021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.191177 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.191156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.191220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.191166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.191655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.191632    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a2902f-a1ff-4ac0-9079-809421c69a6a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.193350 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.193326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.193768 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.193744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-dshm\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.198473 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.198443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjfw5\" (UniqueName: \"kubernetes.io/projected/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kube-api-access-gjfw5\") pod \"tokenizer-migration-path-test-kserve-9d47589cc-hgdpb\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.238127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.238046    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"]
Sep 17 19:28:44.242897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.242876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.260062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.260037    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"]
Sep 17 19:28:44.291309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.291281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abd911e-b348-41d4-bb90-1333d516efd2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.291309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.291312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abd911e-b348-41d4-bb90-1333d516efd2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.291629 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.291372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.291629 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.291391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.302839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.302813    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:44.392380 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.392345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.392569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.392472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abd911e-b348-41d4-bb90-1333d516efd2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.392569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.392498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abd911e-b348-41d4-bb90-1333d516efd2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.392569 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.392557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.392898 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.392869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.392898 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.392889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.393193 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.393162    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abd911e-b348-41d4-bb90-1333d516efd2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.395074 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.395054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abd911e-b348-41d4-bb90-1333d516efd2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.556006 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.555971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:44.581665 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.581635    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"]
Sep 17 19:28:44.583770 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:28:44.583729    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a2902f_a1ff_4ac0_9079_809421c69a6a.slice/crio-e6ea4d1033db24a7fd9df37ef4dc73e891a53ea2647b7bf00dc537b821abbdcb WatchSource:0}: Error finding container e6ea4d1033db24a7fd9df37ef4dc73e891a53ea2647b7bf00dc537b821abbdcb: Status 404 returned error can't find the container with id e6ea4d1033db24a7fd9df37ef4dc73e891a53ea2647b7bf00dc537b821abbdcb
Sep 17 19:28:44.842719 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:44.842661    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"]
Sep 17 19:28:44.843039 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:28:44.843011    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abd911e_b348_41d4_bb90_1333d516efd2.slice/crio-ae7fe269313bee3b02960b98a6ab154118c912bdfdabd614746cceed47d8c244 WatchSource:0}: Error finding container ae7fe269313bee3b02960b98a6ab154118c912bdfdabd614746cceed47d8c244: Status 404 returned error can't find the container with id ae7fe269313bee3b02960b98a6ab154118c912bdfdabd614746cceed47d8c244
Sep 17 19:28:45.552272 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:45.552225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" event={"ID":"e6a2902f-a1ff-4ac0-9079-809421c69a6a","Type":"ContainerStarted","Data":"7eba9ef86851c7d3901a2d85acd229de0bcc16b8a7f80030c665480b19ef80ba"}
Sep 17 19:28:45.552778 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:45.552300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" event={"ID":"e6a2902f-a1ff-4ac0-9079-809421c69a6a","Type":"ContainerStarted","Data":"e6ea4d1033db24a7fd9df37ef4dc73e891a53ea2647b7bf00dc537b821abbdcb"}
Sep 17 19:28:45.554245 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:45.554211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" event={"ID":"7abd911e-b348-41d4-bb90-1333d516efd2","Type":"ContainerStarted","Data":"9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe"}
Sep 17 19:28:45.554384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:45.554258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" event={"ID":"7abd911e-b348-41d4-bb90-1333d516efd2","Type":"ContainerStarted","Data":"ae7fe269313bee3b02960b98a6ab154118c912bdfdabd614746cceed47d8c244"}
Sep 17 19:28:47.572788 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:47.572758    2583 generic.go:356] "Generic (PLEG): container finished" podID="7abd911e-b348-41d4-bb90-1333d516efd2" containerID="9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe" exitCode=0
Sep 17 19:28:47.573161 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:47.572832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" event={"ID":"7abd911e-b348-41d4-bb90-1333d516efd2","Type":"ContainerDied","Data":"9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe"}
Sep 17 19:28:47.574766 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:47.574746    2583 generic.go:356] "Generic (PLEG): container finished" podID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerID="7eba9ef86851c7d3901a2d85acd229de0bcc16b8a7f80030c665480b19ef80ba" exitCode=0
Sep 17 19:28:47.574867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:47.574823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" event={"ID":"e6a2902f-a1ff-4ac0-9079-809421c69a6a","Type":"ContainerDied","Data":"7eba9ef86851c7d3901a2d85acd229de0bcc16b8a7f80030c665480b19ef80ba"}
Sep 17 19:28:48.583096 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:48.583054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" event={"ID":"e6a2902f-a1ff-4ac0-9079-809421c69a6a","Type":"ContainerStarted","Data":"d338a07b08eae2aca9f883e1c7bca68b259d3e52256c7c3db3334d357e9023f7"}
Sep 17 19:28:48.585231 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:48.585203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" event={"ID":"7abd911e-b348-41d4-bb90-1333d516efd2","Type":"ContainerStarted","Data":"a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52"}
Sep 17 19:28:48.585433 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:48.585420    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:28:48.610029 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:48.609977    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" podStartSLOduration=2.9943445950000003 podStartE2EDuration="5.609958203s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.615613608s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:48.606735971 +0000 UTC m=+1330.654615829" watchObservedRunningTime="2026-09-17 19:28:48.609958203 +0000 UTC m=+1330.657838062"
Sep 17 19:28:48.627985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:48.627936    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" podStartSLOduration=2.064778638 podStartE2EDuration="4.627921105s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.563142467s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:48.625242874 +0000 UTC m=+1330.673122756" watchObservedRunningTime="2026-09-17 19:28:48.627921105 +0000 UTC m=+1330.675800962"
Sep 17 19:28:49.303142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:49.303100    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:50.481946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:50.481895    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:28:54.303623 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:54.303577    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:54.323093 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:54.323067    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:28:54.635597 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:28:54.635513    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:29:00.481497 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:00.481442    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:29:10.481267 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:10.481231    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:29:19.601366 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:19.601332    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:29:20.481815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:20.481778    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:29:21.130137 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.130101    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p_7abd911e-b348-41d4-bb90-1333d516efd2/main/0.log"
Sep 17 19:29:21.502440 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.502354    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"]
Sep 17 19:29:21.502957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.502918    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" podUID="7abd911e-b348-41d4-bb90-1333d516efd2" containerName="main" containerID="cri-o://a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52" gracePeriod=30
Sep 17 19:29:21.507527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.507497    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"]
Sep 17 19:29:21.517993 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:29:21.517963    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:29:21.517993 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:21.517993 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:21.517993 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="d338a07b08eae2aca9f883e1c7bca68b259d3e52256c7c3db3334d357e9023f7" cmd=["/bin/sleep","15"]
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.518013    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" message="" err=<
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:29:21.518030    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="main" containerID="cri-o://d338a07b08eae2aca9f883e1c7bca68b259d3e52256c7c3db3334d357e9023f7"
Sep 17 19:29:21.518234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.518061    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="main" containerID="cri-o://d338a07b08eae2aca9f883e1c7bca68b259d3e52256c7c3db3334d357e9023f7" gracePeriod=60
Sep 17 19:29:21.624391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.624268    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 19:29:21.799039 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.798838    2583 generic.go:356] "Generic (PLEG): container finished" podID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerID="d338a07b08eae2aca9f883e1c7bca68b259d3e52256c7c3db3334d357e9023f7" exitCode=0
Sep 17 19:29:21.799039 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.798916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" event={"ID":"e6a2902f-a1ff-4ac0-9079-809421c69a6a","Type":"ContainerDied","Data":"d338a07b08eae2aca9f883e1c7bca68b259d3e52256c7c3db3334d357e9023f7"}
Sep 17 19:29:21.874170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:21.874141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:29:22.040905 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.040839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a2902f-a1ff-4ac0-9079-809421c69a6a-cabundle-cert\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.040909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tmp-dir\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.040936    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kserve-provision-location\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.040953    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-model-cache\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041241 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041220    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.041241 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.041359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a2902f-a1ff-4ac0-9079-809421c69a6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:22.041359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041300    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-home\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041469 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041388    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-dshm\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041469 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041423    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tls-certs\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041469 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjfw5\" (UniqueName: \"kubernetes.io/projected/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kube-api-access-gjfw5\") pod \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\" (UID: \"e6a2902f-a1ff-4ac0-9079-809421c69a6a\") "
Sep 17 19:29:22.041626 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041468    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-home" (OuterVolumeSpecName: "home") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.042043 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041858    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6a2902f-a1ff-4ac0-9079-809421c69a6a-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.042043 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041882    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.042043 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041895    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.042043 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.041906    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.044633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.044599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-dshm" (OuterVolumeSpecName: "dshm") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.044855 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.044826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:22.045356 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.045328    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kube-api-access-gjfw5" (OuterVolumeSpecName: "kube-api-access-gjfw5") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "kube-api-access-gjfw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:22.066154 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.066100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6a2902f-a1ff-4ac0-9079-809421c69a6a" (UID: "e6a2902f-a1ff-4ac0-9079-809421c69a6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.142670 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.142640    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.142670 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.142668    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a2902f-a1ff-4ac0-9079-809421c69a6a-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.143116 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.142680    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a2902f-a1ff-4ac0-9079-809421c69a6a-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.143116 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.142709    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjfw5\" (UniqueName: \"kubernetes.io/projected/e6a2902f-a1ff-4ac0-9079-809421c69a6a-kube-api-access-gjfw5\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.159916 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.159895    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:29:22.344945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.344916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abd911e-b348-41d4-bb90-1333d516efd2-cabundle-cert\") pod \"7abd911e-b348-41d4-bb90-1333d516efd2\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") "
Sep 17 19:29:22.345092 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.344960    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abd911e-b348-41d4-bb90-1333d516efd2-tls-certs\") pod \"7abd911e-b348-41d4-bb90-1333d516efd2\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") "
Sep 17 19:29:22.345092 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.345008    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-tmp-dir\") pod \"7abd911e-b348-41d4-bb90-1333d516efd2\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") "
Sep 17 19:29:22.345092 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.345040    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-kserve-provision-location\") pod \"7abd911e-b348-41d4-bb90-1333d516efd2\" (UID: \"7abd911e-b348-41d4-bb90-1333d516efd2\") "
Sep 17 19:29:22.345309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.345283    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abd911e-b348-41d4-bb90-1333d516efd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7abd911e-b348-41d4-bb90-1333d516efd2" (UID: "7abd911e-b348-41d4-bb90-1333d516efd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:22.345465 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.345428    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7abd911e-b348-41d4-bb90-1333d516efd2" (UID: "7abd911e-b348-41d4-bb90-1333d516efd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.347354 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.347332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abd911e-b348-41d4-bb90-1333d516efd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7abd911e-b348-41d4-bb90-1333d516efd2" (UID: "7abd911e-b348-41d4-bb90-1333d516efd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:22.378943 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.378916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7abd911e-b348-41d4-bb90-1333d516efd2" (UID: "7abd911e-b348-41d4-bb90-1333d516efd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:22.446039 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.446016    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abd911e-b348-41d4-bb90-1333d516efd2-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.446039 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.446039    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abd911e-b348-41d4-bb90-1333d516efd2-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.446203 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.446047    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.446203 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.446055    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abd911e-b348-41d4-bb90-1333d516efd2-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:29:22.806700 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.806662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"
Sep 17 19:29:22.806883 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.806713    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb" event={"ID":"e6a2902f-a1ff-4ac0-9079-809421c69a6a","Type":"ContainerDied","Data":"e6ea4d1033db24a7fd9df37ef4dc73e891a53ea2647b7bf00dc537b821abbdcb"}
Sep 17 19:29:22.806883 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.806784    2583 scope.go:117] "RemoveContainer" containerID="d338a07b08eae2aca9f883e1c7bca68b259d3e52256c7c3db3334d357e9023f7"
Sep 17 19:29:22.809477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.809448    2583 generic.go:356] "Generic (PLEG): container finished" podID="7abd911e-b348-41d4-bb90-1333d516efd2" containerID="a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52" exitCode=0
Sep 17 19:29:22.809583 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.809487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" event={"ID":"7abd911e-b348-41d4-bb90-1333d516efd2","Type":"ContainerDied","Data":"a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52"}
Sep 17 19:29:22.809583 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.809515    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"
Sep 17 19:29:22.809583 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.809531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p" event={"ID":"7abd911e-b348-41d4-bb90-1333d516efd2","Type":"ContainerDied","Data":"ae7fe269313bee3b02960b98a6ab154118c912bdfdabd614746cceed47d8c244"}
Sep 17 19:29:22.820983 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.819422    2583 scope.go:117] "RemoveContainer" containerID="7eba9ef86851c7d3901a2d85acd229de0bcc16b8a7f80030c665480b19ef80ba"
Sep 17 19:29:22.829847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.829827    2583 scope.go:117] "RemoveContainer" containerID="a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52"
Sep 17 19:29:22.831172 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.831136    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"]
Sep 17 19:29:22.838142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.838099    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d47589cc-hgdpb"]
Sep 17 19:29:22.838677 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.838659    2583 scope.go:117] "RemoveContainer" containerID="9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe"
Sep 17 19:29:22.847427 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.847406    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"]
Sep 17 19:29:22.848634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.848619    2583 scope.go:117] "RemoveContainer" containerID="a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52"
Sep 17 19:29:22.848926 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:29:22.848910    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52\": container with ID starting with a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52 not found: ID does not exist" containerID="a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52"
Sep 17 19:29:22.848979 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.848936    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52"} err="failed to get container status \"a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52\": rpc error: code = NotFound desc = could not find container \"a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52\": container with ID starting with a58fbe90f29d11d5d654305aba4f901a8228555ff3c8b1a5fed2319ff9c59b52 not found: ID does not exist"
Sep 17 19:29:22.848979 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.848952    2583 scope.go:117] "RemoveContainer" containerID="9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe"
Sep 17 19:29:22.849234 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:29:22.849213    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe\": container with ID starting with 9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe not found: ID does not exist" containerID="9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe"
Sep 17 19:29:22.849295 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.849244    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe"} err="failed to get container status \"9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe\": rpc error: code = NotFound desc = could not find container \"9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe\": container with ID starting with 9123c9ede98296d425c878dd0310928518f78a4d33149727698be7b60fd1fbfe not found: ID does not exist"
Sep 17 19:29:22.852978 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:22.852956    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-75545d5c5c-khg2p"]
Sep 17 19:29:24.256896 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:24.256857    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abd911e-b348-41d4-bb90-1333d516efd2" path="/var/lib/kubelet/pods/7abd911e-b348-41d4-bb90-1333d516efd2/volumes"
Sep 17 19:29:24.257364 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:24.257320    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" path="/var/lib/kubelet/pods/e6a2902f-a1ff-4ac0-9079-809421c69a6a/volumes"
Sep 17 19:29:30.481820 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:30.481777    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:29:40.481995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:40.481951    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:29:50.481463 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:29:50.481419    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:30:00.481735 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:00.481666    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 17 19:30:10.491548 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:10.491515    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:30:10.499952 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:10.499918    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:30:19.678703 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.678602    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"]
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679008    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="storage-initializer"
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679021    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="storage-initializer"
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679028    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abd911e-b348-41d4-bb90-1333d516efd2" containerName="main"
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679033    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abd911e-b348-41d4-bb90-1333d516efd2" containerName="main"
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679044    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abd911e-b348-41d4-bb90-1333d516efd2" containerName="storage-initializer"
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679049    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abd911e-b348-41d4-bb90-1333d516efd2" containerName="storage-initializer"
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679058    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="main"
Sep 17 19:30:19.679066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679063    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="main"
Sep 17 19:30:19.679350 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679127    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abd911e-b348-41d4-bb90-1333d516efd2" containerName="main"
Sep 17 19:30:19.679350 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.679133    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6a2902f-a1ff-4ac0-9079-809421c69a6a" containerName="main"
Sep 17 19:30:19.682991 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.682970    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.685877 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.685857    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-p8278\""
Sep 17 19:30:19.686857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.686839    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 19:30:19.686924 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.686872    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 19:30:19.687146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.687130    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:30:19.687202 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.687173    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 19:30:19.687284 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.687202    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 19:30:19.697667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.697647    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"]
Sep 17 19:30:19.843618 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.843618 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d9a58a-8381-4f56-8fbd-4872ae840d85-tls-certs\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.843908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-model-cache\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.843908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-home\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.843908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5d6f\" (UniqueName: \"kubernetes.io/projected/83d9a58a-8381-4f56-8fbd-4872ae840d85-kube-api-access-c5d6f\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.843908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-dshm\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.843908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-tmp-dir\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.843908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.843898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d9a58a-8381-4f56-8fbd-4872ae840d85-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945083 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-dshm\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945083 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-tmp-dir\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945083 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d9a58a-8381-4f56-8fbd-4872ae840d85-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d9a58a-8381-4f56-8fbd-4872ae840d85-tls-certs\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-model-cache\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-home\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5d6f\" (UniqueName: \"kubernetes.io/projected/83d9a58a-8381-4f56-8fbd-4872ae840d85-kube-api-access-c5d6f\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-tmp-dir\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-model-cache\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-home\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.945947 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.945896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d9a58a-8381-4f56-8fbd-4872ae840d85-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.947407 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.947380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-dshm\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.947779 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.947761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d9a58a-8381-4f56-8fbd-4872ae840d85-tls-certs\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.956151 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.956105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5d6f\" (UniqueName: \"kubernetes.io/projected/83d9a58a-8381-4f56-8fbd-4872ae840d85-kube-api-access-c5d6f\") pod \"tokenizer-template-args-test-kserve-8b55465f5-49w9b\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:19.976567 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.976538    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"]
Sep 17 19:30:19.981711 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.981627    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:19.993604 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.993581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"]
Sep 17 19:30:19.993797 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:19.993777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:20.147832 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.147802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc088396-81e1-4116-9a5a-00fb7e862fc6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.147982 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.147838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.147982 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.147902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.148055 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.147988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc088396-81e1-4116-9a5a-00fb7e862fc6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.248846 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.248762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc088396-81e1-4116-9a5a-00fb7e862fc6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.248846 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.248807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.249051 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.248864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.249051 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.248919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc088396-81e1-4116-9a5a-00fb7e862fc6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.249532 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.249474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.249532 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.249488    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.249756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.249594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc088396-81e1-4116-9a5a-00fb7e862fc6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.252095 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.252068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc088396-81e1-4116-9a5a-00fb7e862fc6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.280040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.280016    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"]
Sep 17 19:30:20.281297 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:30:20.281274    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d9a58a_8381_4f56_8fbd_4872ae840d85.slice/crio-af71924bfe1dc4df5a89c544f8abc769e248d4896c91f31bccf33ee8b2d4c8c9 WatchSource:0}: Error finding container af71924bfe1dc4df5a89c544f8abc769e248d4896c91f31bccf33ee8b2d4c8c9: Status 404 returned error can't find the container with id af71924bfe1dc4df5a89c544f8abc769e248d4896c91f31bccf33ee8b2d4c8c9
Sep 17 19:30:20.295412 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.295395    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:20.572568 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:20.572539    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"]
Sep 17 19:30:20.573224 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:30:20.573198    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc088396_81e1_4116_9a5a_00fb7e862fc6.slice/crio-21bf46294fb2965b1af45a08f643f889cd3f09920198b5b90e8d56a3b0587b52 WatchSource:0}: Error finding container 21bf46294fb2965b1af45a08f643f889cd3f09920198b5b90e8d56a3b0587b52: Status 404 returned error can't find the container with id 21bf46294fb2965b1af45a08f643f889cd3f09920198b5b90e8d56a3b0587b52
Sep 17 19:30:21.148920 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:21.148883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" event={"ID":"83d9a58a-8381-4f56-8fbd-4872ae840d85","Type":"ContainerStarted","Data":"36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d"}
Sep 17 19:30:21.149340 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:21.148953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" event={"ID":"83d9a58a-8381-4f56-8fbd-4872ae840d85","Type":"ContainerStarted","Data":"af71924bfe1dc4df5a89c544f8abc769e248d4896c91f31bccf33ee8b2d4c8c9"}
Sep 17 19:30:21.150654 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:21.150620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" event={"ID":"fc088396-81e1-4116-9a5a-00fb7e862fc6","Type":"ContainerStarted","Data":"a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e"}
Sep 17 19:30:21.150803 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:21.150682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" event={"ID":"fc088396-81e1-4116-9a5a-00fb7e862fc6","Type":"ContainerStarted","Data":"21bf46294fb2965b1af45a08f643f889cd3f09920198b5b90e8d56a3b0587b52"}
Sep 17 19:30:22.785373 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:22.785333    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"]
Sep 17 19:30:23.168522 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:23.168478    2583 generic.go:356] "Generic (PLEG): container finished" podID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerID="36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d" exitCode=0
Sep 17 19:30:23.168710 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:23.168546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" event={"ID":"83d9a58a-8381-4f56-8fbd-4872ae840d85","Type":"ContainerDied","Data":"36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d"}
Sep 17 19:30:24.175586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:24.175550    2583 generic.go:356] "Generic (PLEG): container finished" podID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerID="a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e" exitCode=0
Sep 17 19:30:24.176016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:24.175627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" event={"ID":"fc088396-81e1-4116-9a5a-00fb7e862fc6","Type":"ContainerDied","Data":"a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e"}
Sep 17 19:30:24.177708 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:24.177670    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" event={"ID":"83d9a58a-8381-4f56-8fbd-4872ae840d85","Type":"ContainerStarted","Data":"3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959"}
Sep 17 19:30:24.216946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:24.216887    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" podStartSLOduration=2.655931047 podStartE2EDuration="5.216873672s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.560942625s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:24.215914187 +0000 UTC m=+1426.263794062" watchObservedRunningTime="2026-09-17 19:30:24.216873672 +0000 UTC m=+1426.264753528"
Sep 17 19:30:24.993941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:24.993909    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:25.186141 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:25.186088    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" event={"ID":"fc088396-81e1-4116-9a5a-00fb7e862fc6","Type":"ContainerStarted","Data":"dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53"}
Sep 17 19:30:25.186583 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:25.186463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:25.207253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:25.207196    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" podStartSLOduration=3.659004893 podStartE2EDuration="6.20717804s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.548173147s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:25.204939061 +0000 UTC m=+1427.252818951" watchObservedRunningTime="2026-09-17 19:30:25.20717804 +0000 UTC m=+1427.255057897"
Sep 17 19:30:29.994586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:29.994553    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:30.007734 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:30.007709    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:30.225350 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:30.225317    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:37.813146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:37.813104    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" containerID="cri-o://dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc" gracePeriod=45
Sep 17 19:30:38.564893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.564858    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:30:38.623501 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623422    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-home\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623503    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ab363-1673-415c-a400-02f52f14ef74-tls-certs\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-model-cache\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623585    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-dshm\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zchm7\" (UniqueName: \"kubernetes.io/projected/ca3ab363-1673-415c-a400-02f52f14ef74-kube-api-access-zchm7\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623659    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-tmp-dir\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623719    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-kserve-provision-location\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623797    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca3ab363-1673-415c-a400-02f52f14ef74-cabundle-cert\") pod \"ca3ab363-1673-415c-a400-02f52f14ef74\" (UID: \"ca3ab363-1673-415c-a400-02f52f14ef74\") "
Sep 17 19:30:38.623897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.623849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:38.624112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.624049    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-home" (OuterVolumeSpecName: "home") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:38.624112 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.624076    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:38.624538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.624498    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca3ab363-1673-415c-a400-02f52f14ef74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:38.626258 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.626208    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca3ab363-1673-415c-a400-02f52f14ef74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:38.626793 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.626754    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-dshm" (OuterVolumeSpecName: "dshm") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:38.626994 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.626969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3ab363-1673-415c-a400-02f52f14ef74-kube-api-access-zchm7" (OuterVolumeSpecName: "kube-api-access-zchm7") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "kube-api-access-zchm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:38.637405 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.637381    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:38.652321 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.652293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca3ab363-1673-415c-a400-02f52f14ef74" (UID: "ca3ab363-1673-415c-a400-02f52f14ef74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:38.724476 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.724452    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:38.724476 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.724475    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zchm7\" (UniqueName: \"kubernetes.io/projected/ca3ab363-1673-415c-a400-02f52f14ef74-kube-api-access-zchm7\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:38.724607 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.724487    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:38.724607 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.724495    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:38.724607 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.724503    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca3ab363-1673-415c-a400-02f52f14ef74-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:38.724607 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.724511    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca3ab363-1673-415c-a400-02f52f14ef74-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:38.724607 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:38.724518    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca3ab363-1673-415c-a400-02f52f14ef74-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:39.269587 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.269550    2583 generic.go:356] "Generic (PLEG): container finished" podID="ca3ab363-1673-415c-a400-02f52f14ef74" containerID="dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc" exitCode=0
Sep 17 19:30:39.270086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.269610    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"
Sep 17 19:30:39.270086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.269638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" event={"ID":"ca3ab363-1673-415c-a400-02f52f14ef74","Type":"ContainerDied","Data":"dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc"}
Sep 17 19:30:39.270086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.269737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" event={"ID":"ca3ab363-1673-415c-a400-02f52f14ef74","Type":"ContainerDied","Data":"4ca210a7f75b2557c404e213266b1b3983f2b452f6a10c59b3af6cd2d43edebb"}
Sep 17 19:30:39.270086 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.269779    2583 scope.go:117] "RemoveContainer" containerID="dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc"
Sep 17 19:30:39.281444 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.281429    2583 scope.go:117] "RemoveContainer" containerID="b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c"
Sep 17 19:30:39.291457 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.291432    2583 scope.go:117] "RemoveContainer" containerID="dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc"
Sep 17 19:30:39.291803 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:39.291774    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc\": container with ID starting with dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc not found: ID does not exist" containerID="dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc"
Sep 17 19:30:39.291902 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.291816    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc"} err="failed to get container status \"dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc\": rpc error: code = NotFound desc = could not find container \"dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc\": container with ID starting with dc39723dea9ad3add1fe91465d08e35cc562681749f94139712c266c97edfdfc not found: ID does not exist"
Sep 17 19:30:39.291902 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.291835    2583 scope.go:117] "RemoveContainer" containerID="b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c"
Sep 17 19:30:39.292132 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:39.292111    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c\": container with ID starting with b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c not found: ID does not exist" containerID="b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c"
Sep 17 19:30:39.292229 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.292137    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c"} err="failed to get container status \"b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c\": rpc error: code = NotFound desc = could not find container \"b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c\": container with ID starting with b0b3e35dcdaf973759cf5d7954edc79f635a43c1b7c35af5079d19e2840ce61c not found: ID does not exist"
Sep 17 19:30:39.296420 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.296375    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"]
Sep 17 19:30:39.299465 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.299442    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs"]
Sep 17 19:30:39.493643 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:39.493596    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-58d4c47bbd-9hlqs" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:30:40.256836 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:40.256800    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" path="/var/lib/kubelet/pods/ca3ab363-1673-415c-a400-02f52f14ef74/volumes"
Sep 17 19:30:56.202347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:56.202305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:57.292343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.292316    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk_fc088396-81e1-4116-9a5a-00fb7e862fc6/main/0.log"
Sep 17 19:30:57.523928 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.523896    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"]
Sep 17 19:30:57.531560 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.531529    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"]
Sep 17 19:30:57.531824 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.531775    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" podUID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerName="main" containerID="cri-o://dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53" gracePeriod=30
Sep 17 19:30:57.534644 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:57.534604    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:30:57.534644 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.534644 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.534644 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959" cmd=["/bin/sleep","15"]
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.534669    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" message="" err=<
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:57.534707    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerName="main" containerID="cri-o://3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959"
Sep 17 19:30:57.534889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.534772    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerName="main" containerID="cri-o://3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959" gracePeriod=60
Sep 17 19:30:57.891012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.890950    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:57.995037 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995010    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d9a58a-8381-4f56-8fbd-4872ae840d85-tls-certs\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-model-cache\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995098    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-home\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995291 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995260    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-home" (OuterVolumeSpecName: "home") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.995291 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995260    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5d6f\" (UniqueName: \"kubernetes.io/projected/83d9a58a-8381-4f56-8fbd-4872ae840d85-kube-api-access-c5d6f\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995291 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995278    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-model-cache" (OuterVolumeSpecName: "model-cache") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.995446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995304    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-kserve-provision-location\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-dshm\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d9a58a-8381-4f56-8fbd-4872ae840d85-cabundle-cert\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995401    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-tmp-dir\") pod \"83d9a58a-8381-4f56-8fbd-4872ae840d85\" (UID: \"83d9a58a-8381-4f56-8fbd-4872ae840d85\") "
Sep 17 19:30:57.995667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995647    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:57.995667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995658    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:57.995946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995843    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.995946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.995807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d9a58a-8381-4f56-8fbd-4872ae840d85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:57.998146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.998094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d9a58a-8381-4f56-8fbd-4872ae840d85-kube-api-access-c5d6f" (OuterVolumeSpecName: "kube-api-access-c5d6f") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "kube-api-access-c5d6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:57.998274 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.998144    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-dshm" (OuterVolumeSpecName: "dshm") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:57.998447 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:57.998413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d9a58a-8381-4f56-8fbd-4872ae840d85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:58.016656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.016625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83d9a58a-8381-4f56-8fbd-4872ae840d85" (UID: "83d9a58a-8381-4f56-8fbd-4872ae840d85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.096536 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.096506    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.096673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.096539    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d9a58a-8381-4f56-8fbd-4872ae840d85-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.096673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.096555    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.096673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.096563    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d9a58a-8381-4f56-8fbd-4872ae840d85-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.096673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.096571    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5d6f\" (UniqueName: \"kubernetes.io/projected/83d9a58a-8381-4f56-8fbd-4872ae840d85-kube-api-access-c5d6f\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.096673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.096581    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d9a58a-8381-4f56-8fbd-4872ae840d85-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.169766 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.169738    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:58.298155 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.298128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-tmp-dir\") pod \"fc088396-81e1-4116-9a5a-00fb7e862fc6\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") "
Sep 17 19:30:58.298496 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.298164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc088396-81e1-4116-9a5a-00fb7e862fc6-cabundle-cert\") pod \"fc088396-81e1-4116-9a5a-00fb7e862fc6\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") "
Sep 17 19:30:58.298496 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.298181    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc088396-81e1-4116-9a5a-00fb7e862fc6-tls-certs\") pod \"fc088396-81e1-4116-9a5a-00fb7e862fc6\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") "
Sep 17 19:30:58.298496 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.298290    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-kserve-provision-location\") pod \"fc088396-81e1-4116-9a5a-00fb7e862fc6\" (UID: \"fc088396-81e1-4116-9a5a-00fb7e862fc6\") "
Sep 17 19:30:58.298625 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.298572    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc088396-81e1-4116-9a5a-00fb7e862fc6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc088396-81e1-4116-9a5a-00fb7e862fc6" (UID: "fc088396-81e1-4116-9a5a-00fb7e862fc6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:58.298625 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.298607    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc088396-81e1-4116-9a5a-00fb7e862fc6" (UID: "fc088396-81e1-4116-9a5a-00fb7e862fc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.300484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.300460    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc088396-81e1-4116-9a5a-00fb7e862fc6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc088396-81e1-4116-9a5a-00fb7e862fc6" (UID: "fc088396-81e1-4116-9a5a-00fb7e862fc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:58.322055 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.322029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc088396-81e1-4116-9a5a-00fb7e862fc6" (UID: "fc088396-81e1-4116-9a5a-00fb7e862fc6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.380428 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.380404    2583 generic.go:356] "Generic (PLEG): container finished" podID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerID="3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959" exitCode=0
Sep 17 19:30:58.380539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.380453    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"
Sep 17 19:30:58.380539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.380491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" event={"ID":"83d9a58a-8381-4f56-8fbd-4872ae840d85","Type":"ContainerDied","Data":"3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959"}
Sep 17 19:30:58.380627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.380542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b" event={"ID":"83d9a58a-8381-4f56-8fbd-4872ae840d85","Type":"ContainerDied","Data":"af71924bfe1dc4df5a89c544f8abc769e248d4896c91f31bccf33ee8b2d4c8c9"}
Sep 17 19:30:58.380627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.380571    2583 scope.go:117] "RemoveContainer" containerID="3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959"
Sep 17 19:30:58.382418 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.382396    2583 generic.go:356] "Generic (PLEG): container finished" podID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerID="dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53" exitCode=0
Sep 17 19:30:58.382525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.382445    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"
Sep 17 19:30:58.382645 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.382444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" event={"ID":"fc088396-81e1-4116-9a5a-00fb7e862fc6","Type":"ContainerDied","Data":"dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53"}
Sep 17 19:30:58.382772 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.382674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk" event={"ID":"fc088396-81e1-4116-9a5a-00fb7e862fc6","Type":"ContainerDied","Data":"21bf46294fb2965b1af45a08f643f889cd3f09920198b5b90e8d56a3b0587b52"}
Sep 17 19:30:58.391776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.391760    2583 scope.go:117] "RemoveContainer" containerID="36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d"
Sep 17 19:30:58.399480 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.399456    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.399574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.399482    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc088396-81e1-4116-9a5a-00fb7e862fc6-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.399574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.399497    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc088396-81e1-4116-9a5a-00fb7e862fc6-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.399574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.399509    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc088396-81e1-4116-9a5a-00fb7e862fc6-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.399984 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.399913    2583 scope.go:117] "RemoveContainer" containerID="3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959"
Sep 17 19:30:58.400227 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:58.400204    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959\": container with ID starting with 3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959 not found: ID does not exist" containerID="3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959"
Sep 17 19:30:58.400333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.400231    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959"} err="failed to get container status \"3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959\": rpc error: code = NotFound desc = could not find container \"3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959\": container with ID starting with 3b33be538de01f78079a9950b41e3a36620c8b2b73e47dcf3120219e7efa9959 not found: ID does not exist"
Sep 17 19:30:58.400333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.400247    2583 scope.go:117] "RemoveContainer" containerID="36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d"
Sep 17 19:30:58.400523 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:58.400507    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d\": container with ID starting with 36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d not found: ID does not exist" containerID="36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d"
Sep 17 19:30:58.400587 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.400525    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d"} err="failed to get container status \"36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d\": rpc error: code = NotFound desc = could not find container \"36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d\": container with ID starting with 36fed2cfa5a00242bb0e339237746a81449028ba1f2368d5a60d70ab15eef91d not found: ID does not exist"
Sep 17 19:30:58.400587 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.400537    2583 scope.go:117] "RemoveContainer" containerID="dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53"
Sep 17 19:30:58.405886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.405865    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"]
Sep 17 19:30:58.410118 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.410097    2583 scope.go:117] "RemoveContainer" containerID="a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e"
Sep 17 19:30:58.416053 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.415997    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-8b55465f5-49w9b"]
Sep 17 19:30:58.423329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.423306    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"]
Sep 17 19:30:58.424440 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.424427    2583 scope.go:117] "RemoveContainer" containerID="dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53"
Sep 17 19:30:58.424879 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:58.424851    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53\": container with ID starting with dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53 not found: ID does not exist" containerID="dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53"
Sep 17 19:30:58.424981 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.424914    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53"} err="failed to get container status \"dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53\": rpc error: code = NotFound desc = could not find container \"dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53\": container with ID starting with dc20401373ed83df90b95df66c037acbfc0b6efcc856ca50af6533288b4b9f53 not found: ID does not exist"
Sep 17 19:30:58.424981 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.424964    2583 scope.go:117] "RemoveContainer" containerID="a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e"
Sep 17 19:30:58.425379 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:30:58.425353    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e\": container with ID starting with a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e not found: ID does not exist" containerID="a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e"
Sep 17 19:30:58.425487 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.425386    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e"} err="failed to get container status \"a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e\": rpc error: code = NotFound desc = could not find container \"a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e\": container with ID starting with a483604b488674c74bca1086844636bae6176d139df9433cd3fbda61428b599e not found: ID does not exist"
Sep 17 19:30:58.427426 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:30:58.427408    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5c4fc6887c-td9pk"]
Sep 17 19:31:00.255224 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:00.255186    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" path="/var/lib/kubelet/pods/83d9a58a-8381-4f56-8fbd-4872ae840d85/volumes"
Sep 17 19:31:00.255634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:00.255618    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc088396-81e1-4116-9a5a-00fb7e862fc6" path="/var/lib/kubelet/pods/fc088396-81e1-4116-9a5a-00fb7e862fc6/volumes"
Sep 17 19:31:26.951932 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.951891    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"]
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952255    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerName="storage-initializer"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952266    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerName="storage-initializer"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952275    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952280    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952290    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="storage-initializer"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952295    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="storage-initializer"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952307    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerName="storage-initializer"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952312    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerName="storage-initializer"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952318    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952322    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952336    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952341    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952396    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d9a58a-8381-4f56-8fbd-4872ae840d85" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952407    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca3ab363-1673-415c-a400-02f52f14ef74" containerName="main"
Sep 17 19:31:26.952429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.952413    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc088396-81e1-4116-9a5a-00fb7e862fc6" containerName="main"
Sep 17 19:31:26.955781 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.955761    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.958483 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.958456    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 19:31:26.958617 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.958497    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 19:31:26.958617 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.958530    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:31:26.958617 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.958561    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-2bbfj\""
Sep 17 19:31:26.958617 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.958497    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 19:31:26.958917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.958804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 19:31:26.969935 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.969910    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"]
Sep 17 19:31:26.994786 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.994759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcqv\" (UniqueName: \"kubernetes.io/projected/dd9b767d-5004-4c98-bd57-790cd1762e22-kube-api-access-9hcqv\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.994899 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.994802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-model-cache\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.994899 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.994830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-tmp-dir\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.995062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.994951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-dshm\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.995062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.995004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b767d-5004-4c98-bd57-790cd1762e22-tls-certs\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.995062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.995035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9b767d-5004-4c98-bd57-790cd1762e22-cabundle-cert\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.995327 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.995078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-home\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:26.995327 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:26.995105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-kserve-provision-location\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.095969 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.095947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-dshm\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.096127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.095983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b767d-5004-4c98-bd57-790cd1762e22-tls-certs\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.096127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.096010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9b767d-5004-4c98-bd57-790cd1762e22-cabundle-cert\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.096127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.096041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-home\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.096127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.096061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-kserve-provision-location\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.096127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.096085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcqv\" (UniqueName: \"kubernetes.io/projected/dd9b767d-5004-4c98-bd57-790cd1762e22-kube-api-access-9hcqv\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.096127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.096115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-model-cache\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.096423 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.096175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-tmp-dir\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.097454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.097431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-home\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.097555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.097472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-tmp-dir\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.097555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.097521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-model-cache\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.097555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.097533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-kserve-provision-location\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.097916 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.097893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9b767d-5004-4c98-bd57-790cd1762e22-cabundle-cert\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.098738 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.098718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-dshm\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.100271 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.100253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b767d-5004-4c98-bd57-790cd1762e22-tls-certs\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.104819 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.104791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcqv\" (UniqueName: \"kubernetes.io/projected/dd9b767d-5004-4c98-bd57-790cd1762e22-kube-api-access-9hcqv\") pod \"auth-disabled-test-kserve-6cc97cc65c-fznkr\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.266623 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.266542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:27.546963 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:27.546934    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"]
Sep 17 19:31:27.547396 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:31:27.547375    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9b767d_5004_4c98_bd57_790cd1762e22.slice/crio-0ec3165e7c7ea80446835edc02fe77753fac2dd396ebfc6d0378d3a637551476 WatchSource:0}: Error finding container 0ec3165e7c7ea80446835edc02fe77753fac2dd396ebfc6d0378d3a637551476: Status 404 returned error can't find the container with id 0ec3165e7c7ea80446835edc02fe77753fac2dd396ebfc6d0378d3a637551476
Sep 17 19:31:28.554765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:28.554727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" event={"ID":"dd9b767d-5004-4c98-bd57-790cd1762e22","Type":"ContainerStarted","Data":"0601f0a39c29f131e60f683d70ae8a49da5e4aa856d86d804d0943afade107c2"}
Sep 17 19:31:28.555126 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:28.554788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" event={"ID":"dd9b767d-5004-4c98-bd57-790cd1762e22","Type":"ContainerStarted","Data":"0ec3165e7c7ea80446835edc02fe77753fac2dd396ebfc6d0378d3a637551476"}
Sep 17 19:31:30.567824 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:30.567792    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerID="0601f0a39c29f131e60f683d70ae8a49da5e4aa856d86d804d0943afade107c2" exitCode=0
Sep 17 19:31:30.568188 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:30.567852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" event={"ID":"dd9b767d-5004-4c98-bd57-790cd1762e22","Type":"ContainerDied","Data":"0601f0a39c29f131e60f683d70ae8a49da5e4aa856d86d804d0943afade107c2"}
Sep 17 19:31:31.577906 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:31.577867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" event={"ID":"dd9b767d-5004-4c98-bd57-790cd1762e22","Type":"ContainerStarted","Data":"9b802bc2b24f9b07872f15ac28710588817965789cd0ebf18835054dacdec852"}
Sep 17 19:31:31.609850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:31.609802    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podStartSLOduration=3.417573989 podStartE2EDuration="5.609783388s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.192209399s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:31.60690014 +0000 UTC m=+1493.654780000" watchObservedRunningTime="2026-09-17 19:31:31.609783388 +0000 UTC m=+1493.657663245"
Sep 17 19:31:32.267163 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:32.267132    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:37.266891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:37.266853    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:31:37.268633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:37.268598    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:31:47.267722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:47.267663    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:31:53.149120 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.149036    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"]
Sep 17 19:31:53.152192 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.152169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.155952 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.155925    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-lwl79\""
Sep 17 19:31:53.156106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.156003    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 19:31:53.156106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.156098    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 19:31:53.156231 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.156100    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 19:31:53.166024 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.165996    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"]
Sep 17 19:31:53.237035 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.236995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.237035 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.237030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.237277 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.237097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjxd\" (UniqueName: \"kubernetes.io/projected/3710c20a-7818-48aa-8ea5-b8da2ebd792b-kube-api-access-8wjxd\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.237277 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.237129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.237277 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.237180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.237277 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.237198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338276 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjxd\" (UniqueName: \"kubernetes.io/projected/3710c20a-7818-48aa-8ea5-b8da2ebd792b-kube-api-access-8wjxd\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338778 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.338778 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.339034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.338999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.340834 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.340809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.341324 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.341304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.346625 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.346600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjxd\" (UniqueName: \"kubernetes.io/projected/3710c20a-7818-48aa-8ea5-b8da2ebd792b-kube-api-access-8wjxd\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.420251 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.420171    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"]
Sep 17 19:31:53.423591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.423561    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.426460 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.426434    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-pmtxr\""
Sep 17 19:31:53.435118 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.435094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"]
Sep 17 19:31:53.464877 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.464860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:53.539969 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.539925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhd92\" (UniqueName: \"kubernetes.io/projected/940e697b-4eff-45ec-9cdc-6328db5da2fa-kube-api-access-qhd92\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.540106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.539982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940e697b-4eff-45ec-9cdc-6328db5da2fa-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.641053 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.641011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhd92\" (UniqueName: \"kubernetes.io/projected/940e697b-4eff-45ec-9cdc-6328db5da2fa-kube-api-access-qhd92\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.641053 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.641048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940e697b-4eff-45ec-9cdc-6328db5da2fa-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.643978 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.643949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940e697b-4eff-45ec-9cdc-6328db5da2fa-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.649454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.649428    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhd92\" (UniqueName: \"kubernetes.io/projected/940e697b-4eff-45ec-9cdc-6328db5da2fa-kube-api-access-qhd92\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.736557 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.736432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:53.743085 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:53.742572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"]
Sep 17 19:31:53.747982 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:31:53.747951    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3710c20a_7818_48aa_8ea5_b8da2ebd792b.slice/crio-6ecb606570cc2aeed9ed700bed44387a782624a4b1202e3f00c3e0c59c90a434 WatchSource:0}: Error finding container 6ecb606570cc2aeed9ed700bed44387a782624a4b1202e3f00c3e0c59c90a434: Status 404 returned error can't find the container with id 6ecb606570cc2aeed9ed700bed44387a782624a4b1202e3f00c3e0c59c90a434
Sep 17 19:31:54.024097 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.024058    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"]
Sep 17 19:31:54.024544 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:31:54.024514    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940e697b_4eff_45ec_9cdc_6328db5da2fa.slice/crio-b92e5d3364d9351556cd45ab92f04dbf2a31a53c3823694893af70ce850dabce WatchSource:0}: Error finding container b92e5d3364d9351556cd45ab92f04dbf2a31a53c3823694893af70ce850dabce: Status 404 returned error can't find the container with id b92e5d3364d9351556cd45ab92f04dbf2a31a53c3823694893af70ce850dabce
Sep 17 19:31:54.724596 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.724546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" event={"ID":"3710c20a-7818-48aa-8ea5-b8da2ebd792b","Type":"ContainerStarted","Data":"c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875"}
Sep 17 19:31:54.725087 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.724621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" event={"ID":"3710c20a-7818-48aa-8ea5-b8da2ebd792b","Type":"ContainerStarted","Data":"6ecb606570cc2aeed9ed700bed44387a782624a4b1202e3f00c3e0c59c90a434"}
Sep 17 19:31:54.726355 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.726327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" event={"ID":"940e697b-4eff-45ec-9cdc-6328db5da2fa","Type":"ContainerStarted","Data":"bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e"}
Sep 17 19:31:54.726476 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.726367    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" event={"ID":"940e697b-4eff-45ec-9cdc-6328db5da2fa","Type":"ContainerStarted","Data":"b92e5d3364d9351556cd45ab92f04dbf2a31a53c3823694893af70ce850dabce"}
Sep 17 19:31:54.726551 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.726471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:31:54.746216 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.746156    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" podStartSLOduration=1.746135154 podStartE2EDuration="1.746135154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:54.742757045 +0000 UTC m=+1516.790636903" watchObservedRunningTime="2026-09-17 19:31:54.746135154 +0000 UTC m=+1516.794015013"
Sep 17 19:31:54.761649 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:54.761604    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" podStartSLOduration=1.761588717 podStartE2EDuration="1.761588717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:31:54.760624635 +0000 UTC m=+1516.808504493" watchObservedRunningTime="2026-09-17 19:31:54.761588717 +0000 UTC m=+1516.809468608"
Sep 17 19:31:55.465394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:55.465360    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:31:57.267280 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:31:57.267238    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:32:03.465747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:03.465678    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:32:03.478375 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:03.478355    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:32:03.790653 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:03.790567    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:32:07.267737 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:07.267673    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:32:17.267030 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:17.266987    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:32:25.735713 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:25.735657    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:32:27.266961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:27.266922    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:32:27.761106 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:27.760988    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"]
Sep 17 19:32:27.765006 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:27.764979    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"]
Sep 17 19:32:27.773597 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:32:27.773564    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:32:27.773597 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:32:27.773597 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:32:27.773597 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875" cmd=["/bin/sleep","15"]
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:27.773711    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" message="" err=<
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:32:27.773736    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" podUID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" containerName="main" containerID="cri-o://c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875"
Sep 17 19:32:27.773931 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:27.773792    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" podUID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" containerName="main" containerID="cri-o://c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875" gracePeriod=60
Sep 17 19:32:27.780323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:27.780290    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" podUID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 17 19:32:28.127625 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.127601    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:32:28.251287 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.251254    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-dshm\") pod \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") "
Sep 17 19:32:28.251462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.251311    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wjxd\" (UniqueName: \"kubernetes.io/projected/3710c20a-7818-48aa-8ea5-b8da2ebd792b-kube-api-access-8wjxd\") pod \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") "
Sep 17 19:32:28.251462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.251339    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-model-cache\") pod \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") "
Sep 17 19:32:28.251462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.251407    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-home\") pod \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") "
Sep 17 19:32:28.251462 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.251460    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tls-certs\") pod \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") "
Sep 17 19:32:28.251733 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.251494    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tmp-dir\") pod \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\" (UID: \"3710c20a-7818-48aa-8ea5-b8da2ebd792b\") "
Sep 17 19:32:28.251921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.251875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-home" (OuterVolumeSpecName: "home") pod "3710c20a-7818-48aa-8ea5-b8da2ebd792b" (UID: "3710c20a-7818-48aa-8ea5-b8da2ebd792b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:28.252104 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.252046    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-model-cache" (OuterVolumeSpecName: "model-cache") pod "3710c20a-7818-48aa-8ea5-b8da2ebd792b" (UID: "3710c20a-7818-48aa-8ea5-b8da2ebd792b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:28.252177 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.252075    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3710c20a-7818-48aa-8ea5-b8da2ebd792b" (UID: "3710c20a-7818-48aa-8ea5-b8da2ebd792b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:28.254088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.254058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3710c20a-7818-48aa-8ea5-b8da2ebd792b-kube-api-access-8wjxd" (OuterVolumeSpecName: "kube-api-access-8wjxd") pod "3710c20a-7818-48aa-8ea5-b8da2ebd792b" (UID: "3710c20a-7818-48aa-8ea5-b8da2ebd792b"). InnerVolumeSpecName "kube-api-access-8wjxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:28.254088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.254048    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-dshm" (OuterVolumeSpecName: "dshm") pod "3710c20a-7818-48aa-8ea5-b8da2ebd792b" (UID: "3710c20a-7818-48aa-8ea5-b8da2ebd792b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:28.254299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.254285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3710c20a-7818-48aa-8ea5-b8da2ebd792b" (UID: "3710c20a-7818-48aa-8ea5-b8da2ebd792b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:28.352554 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.352519    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:32:28.352554 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.352548    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:32:28.353020 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.352561    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjxd\" (UniqueName: \"kubernetes.io/projected/3710c20a-7818-48aa-8ea5-b8da2ebd792b-kube-api-access-8wjxd\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:32:28.353020 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.352573    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:32:28.353020 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.352584    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3710c20a-7818-48aa-8ea5-b8da2ebd792b-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:32:28.353020 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.352594    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3710c20a-7818-48aa-8ea5-b8da2ebd792b-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:32:28.935196 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.935156    2583 generic.go:356] "Generic (PLEG): container finished" podID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" containerID="c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875" exitCode=0
Sep 17 19:32:28.935379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.935201    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"
Sep 17 19:32:28.935379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.935226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" event={"ID":"3710c20a-7818-48aa-8ea5-b8da2ebd792b","Type":"ContainerDied","Data":"c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875"}
Sep 17 19:32:28.935379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.935283    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw" event={"ID":"3710c20a-7818-48aa-8ea5-b8da2ebd792b","Type":"ContainerDied","Data":"6ecb606570cc2aeed9ed700bed44387a782624a4b1202e3f00c3e0c59c90a434"}
Sep 17 19:32:28.935379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.935312    2583 scope.go:117] "RemoveContainer" containerID="c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875"
Sep 17 19:32:28.944727 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.944682    2583 scope.go:117] "RemoveContainer" containerID="c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875"
Sep 17 19:32:28.945024 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:32:28.945005    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875\": container with ID starting with c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875 not found: ID does not exist" containerID="c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875"
Sep 17 19:32:28.945075 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.945032    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875"} err="failed to get container status \"c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875\": rpc error: code = NotFound desc = could not find container \"c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875\": container with ID starting with c19740acde8b6b5f925006c2af09a918bc224547c9bc570b30020de675145875 not found: ID does not exist"
Sep 17 19:32:28.962941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.962911    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"]
Sep 17 19:32:28.974499 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:28.974478    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-rx7nw"]
Sep 17 19:32:30.263066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:30.263023    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" path="/var/lib/kubelet/pods/3710c20a-7818-48aa-8ea5-b8da2ebd792b/volumes"
Sep 17 19:32:35.735760 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:35.735676    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:32:37.267135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:37.267088    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:32:42.787501 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:42.787455    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main" containerID="cri-o://bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e" gracePeriod=45
Sep 17 19:32:45.735815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:45.735774    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:32:47.267282 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:47.267239    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:32:55.735489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:55.735452    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:32:55.735880 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:55.735551    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:32:57.267577 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:32:57.267523    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:33:05.735945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:05.735905    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:07.267868 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:07.267822    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:33:13.043808 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.043779    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:33:13.138339 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.138250    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhd92\" (UniqueName: \"kubernetes.io/projected/940e697b-4eff-45ec-9cdc-6328db5da2fa-kube-api-access-qhd92\") pod \"940e697b-4eff-45ec-9cdc-6328db5da2fa\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") "
Sep 17 19:33:13.138339 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.138298    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940e697b-4eff-45ec-9cdc-6328db5da2fa-tls-certs\") pod \"940e697b-4eff-45ec-9cdc-6328db5da2fa\" (UID: \"940e697b-4eff-45ec-9cdc-6328db5da2fa\") "
Sep 17 19:33:13.140576 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.140536    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940e697b-4eff-45ec-9cdc-6328db5da2fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "940e697b-4eff-45ec-9cdc-6328db5da2fa" (UID: "940e697b-4eff-45ec-9cdc-6328db5da2fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:33:13.140723 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.140602    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940e697b-4eff-45ec-9cdc-6328db5da2fa-kube-api-access-qhd92" (OuterVolumeSpecName: "kube-api-access-qhd92") pod "940e697b-4eff-45ec-9cdc-6328db5da2fa" (UID: "940e697b-4eff-45ec-9cdc-6328db5da2fa"). InnerVolumeSpecName "kube-api-access-qhd92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:33:13.199533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.199495    2583 generic.go:356] "Generic (PLEG): container finished" podID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerID="bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e" exitCode=0
Sep 17 19:33:13.199681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.199533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" event={"ID":"940e697b-4eff-45ec-9cdc-6328db5da2fa","Type":"ContainerDied","Data":"bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e"}
Sep 17 19:33:13.199681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.199556    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"
Sep 17 19:33:13.199681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.199591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv" event={"ID":"940e697b-4eff-45ec-9cdc-6328db5da2fa","Type":"ContainerDied","Data":"b92e5d3364d9351556cd45ab92f04dbf2a31a53c3823694893af70ce850dabce"}
Sep 17 19:33:13.199681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.199616    2583 scope.go:117] "RemoveContainer" containerID="bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e"
Sep 17 19:33:13.208799 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.208773    2583 scope.go:117] "RemoveContainer" containerID="bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e"
Sep 17 19:33:13.209094 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:33:13.209074    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e\": container with ID starting with bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e not found: ID does not exist" containerID="bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e"
Sep 17 19:33:13.209182 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.209102    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e"} err="failed to get container status \"bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e\": rpc error: code = NotFound desc = could not find container \"bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e\": container with ID starting with bd608623996260313006a8740d54a79b89ba00708abe41f965195b36f90ad69e not found: ID does not exist"
Sep 17 19:33:13.226657 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.226629    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"]
Sep 17 19:33:13.233209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.233181    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schews4jv"]
Sep 17 19:33:13.239215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.239192    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhd92\" (UniqueName: \"kubernetes.io/projected/940e697b-4eff-45ec-9cdc-6328db5da2fa-kube-api-access-qhd92\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:13.239281 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:13.239218    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/940e697b-4eff-45ec-9cdc-6328db5da2fa-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:14.256539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:14.256489    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" path="/var/lib/kubelet/pods/940e697b-4eff-45ec-9cdc-6328db5da2fa/volumes"
Sep 17 19:33:17.267420 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:17.267381    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:33:27.276871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:27.276785    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:33:27.285040 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:27.285012    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:33:28.170361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.170329    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"]
Sep 17 19:33:28.170762 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.170748    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main"
Sep 17 19:33:28.170813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.170763    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main"
Sep 17 19:33:28.170813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.170781    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" containerName="main"
Sep 17 19:33:28.170813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.170788    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" containerName="main"
Sep 17 19:33:28.170911 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.170845    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3710c20a-7818-48aa-8ea5-b8da2ebd792b" containerName="main"
Sep 17 19:33:28.170911 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.170851    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="940e697b-4eff-45ec-9cdc-6328db5da2fa" containerName="main"
Sep 17 19:33:28.173811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.173796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.176911 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.176889    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 19:33:28.178080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.178061    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-6b85s\""
Sep 17 19:33:28.178202 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.178080    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 19:33:28.178202 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.178087    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 19:33:28.191439 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.191418    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"]
Sep 17 19:33:28.263302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.263266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.263454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.263357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.263454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.263393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.263454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.263440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.263628 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.263460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.263628 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.263478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4hd\" (UniqueName: \"kubernetes.io/projected/f1e8c299-bf1a-43d8-bc21-f7dc39124409-kube-api-access-9z4hd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364001 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.363970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4hd\" (UniqueName: \"kubernetes.io/projected/f1e8c299-bf1a-43d8-bc21-f7dc39124409-kube-api-access-9z4hd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364394 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364786 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364786 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.364969 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.364948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.366520 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.366498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.366948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.366931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.377609 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.377585    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4hd\" (UniqueName: \"kubernetes.io/projected/f1e8c299-bf1a-43d8-bc21-f7dc39124409-kube-api-access-9z4hd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.378099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.378077    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"]
Sep 17 19:33:28.381944 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.381914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.384957 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.384936    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-rmn57\""
Sep 17 19:33:28.391232 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.391208    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"]
Sep 17 19:33:28.465369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.465301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jbm\" (UniqueName: \"kubernetes.io/projected/9b239dc0-0db1-4f39-8253-2cc3c58aad57-kube-api-access-49jbm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.465369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.465351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239dc0-0db1-4f39-8253-2cc3c58aad57-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.483769 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.483741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:28.566495 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.566409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239dc0-0db1-4f39-8253-2cc3c58aad57-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.566810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.566501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jbm\" (UniqueName: \"kubernetes.io/projected/9b239dc0-0db1-4f39-8253-2cc3c58aad57-kube-api-access-49jbm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.569121 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.569095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239dc0-0db1-4f39-8253-2cc3c58aad57-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.575245 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.575218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jbm\" (UniqueName: \"kubernetes.io/projected/9b239dc0-0db1-4f39-8253-2cc3c58aad57-kube-api-access-49jbm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.701555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.701523    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:28.761290 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.761075    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"]
Sep 17 19:33:28.763537 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:33:28.763515    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e8c299_bf1a_43d8_bc21_f7dc39124409.slice/crio-82ba73af7652818df9e5ed5c4436bcce8b207c9e25ab11569aa57e1820e2f889 WatchSource:0}: Error finding container 82ba73af7652818df9e5ed5c4436bcce8b207c9e25ab11569aa57e1820e2f889: Status 404 returned error can't find the container with id 82ba73af7652818df9e5ed5c4436bcce8b207c9e25ab11569aa57e1820e2f889
Sep 17 19:33:28.977978 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:28.977951    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"]
Sep 17 19:33:28.979083 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:33:28.979056    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b239dc0_0db1_4f39_8253_2cc3c58aad57.slice/crio-d2d2b18ef7083c495aa1c131b5e6874c50a74547f67fb54d19e81ee96754b674 WatchSource:0}: Error finding container d2d2b18ef7083c495aa1c131b5e6874c50a74547f67fb54d19e81ee96754b674: Status 404 returned error can't find the container with id d2d2b18ef7083c495aa1c131b5e6874c50a74547f67fb54d19e81ee96754b674
Sep 17 19:33:29.299773 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.299727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" event={"ID":"9b239dc0-0db1-4f39-8253-2cc3c58aad57","Type":"ContainerStarted","Data":"7f1aae564df2e6d3e784a023c954be237c0cb64b039ec66e694ab5a4a49e9b1c"}
Sep 17 19:33:29.299939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.299788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" event={"ID":"9b239dc0-0db1-4f39-8253-2cc3c58aad57","Type":"ContainerStarted","Data":"d2d2b18ef7083c495aa1c131b5e6874c50a74547f67fb54d19e81ee96754b674"}
Sep 17 19:33:29.300023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.299944    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:33:29.301550 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.301517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" event={"ID":"f1e8c299-bf1a-43d8-bc21-f7dc39124409","Type":"ContainerStarted","Data":"7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f"}
Sep 17 19:33:29.301674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.301569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" event={"ID":"f1e8c299-bf1a-43d8-bc21-f7dc39124409","Type":"ContainerStarted","Data":"82ba73af7652818df9e5ed5c4436bcce8b207c9e25ab11569aa57e1820e2f889"}
Sep 17 19:33:29.320187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.320149    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" podStartSLOduration=1.320136652 podStartE2EDuration="1.320136652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:29.318896311 +0000 UTC m=+1611.366776169" watchObservedRunningTime="2026-09-17 19:33:29.320136652 +0000 UTC m=+1611.368016558"
Sep 17 19:33:29.346236 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.346193    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" podStartSLOduration=1.346179755 podStartE2EDuration="1.346179755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:29.343722777 +0000 UTC m=+1611.391602635" watchObservedRunningTime="2026-09-17 19:33:29.346179755 +0000 UTC m=+1611.394059645"
Sep 17 19:33:29.484747 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:29.484718    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:38.484621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:38.484586    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:38.497014 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:38.496988    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:39.368403 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:39.368376    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:33:43.709545 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:43.709512    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"]
Sep 17 19:33:58.732338 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:58.732285    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" containerID="cri-o://9b802bc2b24f9b07872f15ac28710588817965789cd0ebf18835054dacdec852" gracePeriod=45
Sep 17 19:33:59.278291 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.278251    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 19:33:59.478037 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.477899    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerID="9b802bc2b24f9b07872f15ac28710588817965789cd0ebf18835054dacdec852" exitCode=0
Sep 17 19:33:59.478037 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.477966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" event={"ID":"dd9b767d-5004-4c98-bd57-790cd1762e22","Type":"ContainerDied","Data":"9b802bc2b24f9b07872f15ac28710588817965789cd0ebf18835054dacdec852"}
Sep 17 19:33:59.498380 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.498359    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:33:59.519673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hcqv\" (UniqueName: \"kubernetes.io/projected/dd9b767d-5004-4c98-bd57-790cd1762e22-kube-api-access-9hcqv\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.519673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519654    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-model-cache\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.519817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519710    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-tmp-dir\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.519817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-dshm\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.519817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519776    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-home\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.519817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519810    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b767d-5004-4c98-bd57-790cd1762e22-tls-certs\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.520003 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519885    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-kserve-provision-location\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.520003 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.519940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9b767d-5004-4c98-bd57-790cd1762e22-cabundle-cert\") pod \"dd9b767d-5004-4c98-bd57-790cd1762e22\" (UID: \"dd9b767d-5004-4c98-bd57-790cd1762e22\") "
Sep 17 19:33:59.520121 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.520069    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:59.520305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.520286    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:59.520579 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.520535    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9b767d-5004-4c98-bd57-790cd1762e22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:33:59.520744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.520717    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-home" (OuterVolumeSpecName: "home") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:59.522048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.522024    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9b767d-5004-4c98-bd57-790cd1762e22-kube-api-access-9hcqv" (OuterVolumeSpecName: "kube-api-access-9hcqv") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "kube-api-access-9hcqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:33:59.522196 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.522165    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-dshm" (OuterVolumeSpecName: "dshm") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:59.522264 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.522225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9b767d-5004-4c98-bd57-790cd1762e22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:33:59.535249 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.535222    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:59.546994 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.546966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd9b767d-5004-4c98-bd57-790cd1762e22" (UID: "dd9b767d-5004-4c98-bd57-790cd1762e22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:33:59.620922 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.620898    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:59.620922 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.620920    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:59.621048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.620929    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:59.621048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.620937    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd9b767d-5004-4c98-bd57-790cd1762e22-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:59.621048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.620944    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9b767d-5004-4c98-bd57-790cd1762e22-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:59.621048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.620953    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9b767d-5004-4c98-bd57-790cd1762e22-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:33:59.621048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:33:59.620962    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hcqv\" (UniqueName: \"kubernetes.io/projected/dd9b767d-5004-4c98-bd57-790cd1762e22-kube-api-access-9hcqv\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:34:00.308794 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:00.308768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:34:00.485284 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:00.485256    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"
Sep 17 19:34:00.485435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:00.485282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr" event={"ID":"dd9b767d-5004-4c98-bd57-790cd1762e22","Type":"ContainerDied","Data":"0ec3165e7c7ea80446835edc02fe77753fac2dd396ebfc6d0378d3a637551476"}
Sep 17 19:34:00.485435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:00.485342    2583 scope.go:117] "RemoveContainer" containerID="9b802bc2b24f9b07872f15ac28710588817965789cd0ebf18835054dacdec852"
Sep 17 19:34:00.494925 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:00.494908    2583 scope.go:117] "RemoveContainer" containerID="0601f0a39c29f131e60f683d70ae8a49da5e4aa856d86d804d0943afade107c2"
Sep 17 19:34:00.504914 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:00.504890    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"]
Sep 17 19:34:00.509347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:00.509326    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6cc97cc65c-fznkr"]
Sep 17 19:34:02.255859 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:02.255830    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" path="/var/lib/kubelet/pods/dd9b767d-5004-4c98-bd57-790cd1762e22/volumes"
Sep 17 19:34:05.378499 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.378463    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"]
Sep 17 19:34:05.378921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.378905    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="storage-initializer"
Sep 17 19:34:05.378992 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.378925    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="storage-initializer"
Sep 17 19:34:05.378992 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.378962    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main"
Sep 17 19:34:05.378992 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.378968    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main"
Sep 17 19:34:05.379133 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.379074    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9b767d-5004-4c98-bd57-790cd1762e22" containerName="main"
Sep 17 19:34:05.383909 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.383889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.386741 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.386719    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 19:34:05.395959 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.395935    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"]
Sep 17 19:34:05.470636 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.470608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.470769 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.470640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b761357b-2604-43a1-8c5f-e3996d8d9481-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.470769 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.470734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.470769 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.470754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2hk\" (UniqueName: \"kubernetes.io/projected/b761357b-2604-43a1-8c5f-e3996d8d9481-kube-api-access-xz2hk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.470884 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.470771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.470884 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.470788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b761357b-2604-43a1-8c5f-e3996d8d9481-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2hk\" (UniqueName: \"kubernetes.io/projected/b761357b-2604-43a1-8c5f-e3996d8d9481-kube-api-access-xz2hk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572632 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.572819 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.572654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.574480 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.574461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.574786 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.574771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b761357b-2604-43a1-8c5f-e3996d8d9481-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.581130 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.581108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2hk\" (UniqueName: \"kubernetes.io/projected/b761357b-2604-43a1-8c5f-e3996d8d9481-kube-api-access-xz2hk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.695871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.695805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:05.971545 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:05.971517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"]
Sep 17 19:34:05.972832 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:34:05.972806    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb761357b_2604_43a1_8c5f_e3996d8d9481.slice/crio-359b863a9b9f76fa73ee0e88167a6665605f009ee4b33c0b1684858b71f2559c WatchSource:0}: Error finding container 359b863a9b9f76fa73ee0e88167a6665605f009ee4b33c0b1684858b71f2559c: Status 404 returned error can't find the container with id 359b863a9b9f76fa73ee0e88167a6665605f009ee4b33c0b1684858b71f2559c
Sep 17 19:34:06.527209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:06.527174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" event={"ID":"b761357b-2604-43a1-8c5f-e3996d8d9481","Type":"ContainerStarted","Data":"368b8393cdd779907c979f9c800a67ecf67d575ca36fd2fed96a9a49048bfb73"}
Sep 17 19:34:06.527610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:06.527229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" event={"ID":"b761357b-2604-43a1-8c5f-e3996d8d9481","Type":"ContainerStarted","Data":"359b863a9b9f76fa73ee0e88167a6665605f009ee4b33c0b1684858b71f2559c"}
Sep 17 19:34:06.548090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:06.548040    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" podStartSLOduration=1.548024826 podStartE2EDuration="1.548024826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:06.544827402 +0000 UTC m=+1648.592707260" watchObservedRunningTime="2026-09-17 19:34:06.548024826 +0000 UTC m=+1648.595904682"
Sep 17 19:34:06.696669 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:06.696642    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:15.696495 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:15.696461    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:15.708798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:15.708772    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:16.591588 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:16.591560    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:53.454006 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.453910    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"]
Sep 17 19:34:53.464177 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:34:53.464140    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:34:53.464177 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:53.464177 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:53.464177 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="368b8393cdd779907c979f9c800a67ecf67d575ca36fd2fed96a9a49048bfb73" cmd=["/bin/sleep","15"]
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.464193    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" message="" err=<
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:34:53.464205    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" podUID="b761357b-2604-43a1-8c5f-e3996d8d9481" containerName="main" containerID="cri-o://368b8393cdd779907c979f9c800a67ecf67d575ca36fd2fed96a9a49048bfb73"
Sep 17 19:34:53.464511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.464253    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" podUID="b761357b-2604-43a1-8c5f-e3996d8d9481" containerName="main" containerID="cri-o://368b8393cdd779907c979f9c800a67ecf67d575ca36fd2fed96a9a49048bfb73" gracePeriod=60
Sep 17 19:34:53.581061 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.581029    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" podUID="b761357b-2604-43a1-8c5f-e3996d8d9481" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Sep 17 19:34:53.792603 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.792559    2583 generic.go:356] "Generic (PLEG): container finished" podID="b761357b-2604-43a1-8c5f-e3996d8d9481" containerID="368b8393cdd779907c979f9c800a67ecf67d575ca36fd2fed96a9a49048bfb73" exitCode=0
Sep 17 19:34:53.792781 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.792724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" event={"ID":"b761357b-2604-43a1-8c5f-e3996d8d9481","Type":"ContainerDied","Data":"368b8393cdd779907c979f9c800a67ecf67d575ca36fd2fed96a9a49048bfb73"}
Sep 17 19:34:53.819477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.819453    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:53.961760 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.961724    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b761357b-2604-43a1-8c5f-e3996d8d9481-tls-certs\") pod \"b761357b-2604-43a1-8c5f-e3996d8d9481\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") "
Sep 17 19:34:53.961941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.961772    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-tmp-dir\") pod \"b761357b-2604-43a1-8c5f-e3996d8d9481\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") "
Sep 17 19:34:53.961941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.961840    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-home\") pod \"b761357b-2604-43a1-8c5f-e3996d8d9481\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") "
Sep 17 19:34:53.961941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.961880    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-dshm\") pod \"b761357b-2604-43a1-8c5f-e3996d8d9481\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") "
Sep 17 19:34:53.961941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.961905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-model-cache\") pod \"b761357b-2604-43a1-8c5f-e3996d8d9481\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") "
Sep 17 19:34:53.962174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.961942    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz2hk\" (UniqueName: \"kubernetes.io/projected/b761357b-2604-43a1-8c5f-e3996d8d9481-kube-api-access-xz2hk\") pod \"b761357b-2604-43a1-8c5f-e3996d8d9481\" (UID: \"b761357b-2604-43a1-8c5f-e3996d8d9481\") "
Sep 17 19:34:53.962174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.962022    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b761357b-2604-43a1-8c5f-e3996d8d9481" (UID: "b761357b-2604-43a1-8c5f-e3996d8d9481"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:53.962174 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.962077    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-home" (OuterVolumeSpecName: "home") pod "b761357b-2604-43a1-8c5f-e3996d8d9481" (UID: "b761357b-2604-43a1-8c5f-e3996d8d9481"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:53.962335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.962316    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-model-cache" (OuterVolumeSpecName: "model-cache") pod "b761357b-2604-43a1-8c5f-e3996d8d9481" (UID: "b761357b-2604-43a1-8c5f-e3996d8d9481"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:53.962391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.962375    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:34:53.962444 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.962390    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:34:53.964201 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.964168    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b761357b-2604-43a1-8c5f-e3996d8d9481-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b761357b-2604-43a1-8c5f-e3996d8d9481" (UID: "b761357b-2604-43a1-8c5f-e3996d8d9481"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:53.964332 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.964220    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b761357b-2604-43a1-8c5f-e3996d8d9481-kube-api-access-xz2hk" (OuterVolumeSpecName: "kube-api-access-xz2hk") pod "b761357b-2604-43a1-8c5f-e3996d8d9481" (UID: "b761357b-2604-43a1-8c5f-e3996d8d9481"). InnerVolumeSpecName "kube-api-access-xz2hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:53.964332 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:53.964245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-dshm" (OuterVolumeSpecName: "dshm") pod "b761357b-2604-43a1-8c5f-e3996d8d9481" (UID: "b761357b-2604-43a1-8c5f-e3996d8d9481"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:54.063579 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.063555    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:34:54.063579 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.063579    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b761357b-2604-43a1-8c5f-e3996d8d9481-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:34:54.063754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.063590    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz2hk\" (UniqueName: \"kubernetes.io/projected/b761357b-2604-43a1-8c5f-e3996d8d9481-kube-api-access-xz2hk\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:34:54.063754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.063600    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b761357b-2604-43a1-8c5f-e3996d8d9481-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:34:54.800391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.800354    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"
Sep 17 19:34:54.800896 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.800387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk" event={"ID":"b761357b-2604-43a1-8c5f-e3996d8d9481","Type":"ContainerDied","Data":"359b863a9b9f76fa73ee0e88167a6665605f009ee4b33c0b1684858b71f2559c"}
Sep 17 19:34:54.800896 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.800465    2583 scope.go:117] "RemoveContainer" containerID="368b8393cdd779907c979f9c800a67ecf67d575ca36fd2fed96a9a49048bfb73"
Sep 17 19:34:54.819639 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.819615    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"]
Sep 17 19:34:54.825418 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:54.825396    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-4sgmk"]
Sep 17 19:34:56.255264 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:34:56.255228    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b761357b-2604-43a1-8c5f-e3996d8d9481" path="/var/lib/kubelet/pods/b761357b-2604-43a1-8c5f-e3996d8d9481/volumes"
Sep 17 19:35:43.068173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.068138    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"]
Sep 17 19:35:43.073987 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.073954    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"]
Sep 17 19:35:43.083159 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:35:43.083127    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:35:43.083159 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:35:43.083159 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:35:43.083159 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f" cmd=["/bin/sleep","15"]
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.083176    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" message="" err=<
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:35:43.083188    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" podUID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" containerName="main" containerID="cri-o://7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f"
Sep 17 19:35:43.083451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.083230    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" podUID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" containerName="main" containerID="cri-o://7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f" gracePeriod=60
Sep 17 19:35:43.439583 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.439560    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:35:43.540362 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540334    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-home\") pod \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") "
Sep 17 19:35:43.540517 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540378    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z4hd\" (UniqueName: \"kubernetes.io/projected/f1e8c299-bf1a-43d8-bc21-f7dc39124409-kube-api-access-9z4hd\") pod \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") "
Sep 17 19:35:43.540517 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540422    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-model-cache\") pod \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") "
Sep 17 19:35:43.540517 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540495    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-dshm\") pod \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") "
Sep 17 19:35:43.540725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tls-certs\") pod \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") "
Sep 17 19:35:43.540725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540567    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tmp-dir\") pod \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\" (UID: \"f1e8c299-bf1a-43d8-bc21-f7dc39124409\") "
Sep 17 19:35:43.540725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-home" (OuterVolumeSpecName: "home") pod "f1e8c299-bf1a-43d8-bc21-f7dc39124409" (UID: "f1e8c299-bf1a-43d8-bc21-f7dc39124409"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:43.540880 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540798    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1e8c299-bf1a-43d8-bc21-f7dc39124409" (UID: "f1e8c299-bf1a-43d8-bc21-f7dc39124409"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:43.540939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540872    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.540939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540912    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.540939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.540921    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1e8c299-bf1a-43d8-bc21-f7dc39124409" (UID: "f1e8c299-bf1a-43d8-bc21-f7dc39124409"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:43.542926 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.542897    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1e8c299-bf1a-43d8-bc21-f7dc39124409" (UID: "f1e8c299-bf1a-43d8-bc21-f7dc39124409"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:35:43.542926 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.542907    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e8c299-bf1a-43d8-bc21-f7dc39124409-kube-api-access-9z4hd" (OuterVolumeSpecName: "kube-api-access-9z4hd") pod "f1e8c299-bf1a-43d8-bc21-f7dc39124409" (UID: "f1e8c299-bf1a-43d8-bc21-f7dc39124409"). InnerVolumeSpecName "kube-api-access-9z4hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:35:43.543067 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.542939    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-dshm" (OuterVolumeSpecName: "dshm") pod "f1e8c299-bf1a-43d8-bc21-f7dc39124409" (UID: "f1e8c299-bf1a-43d8-bc21-f7dc39124409"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:43.641542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.641485    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4hd\" (UniqueName: \"kubernetes.io/projected/f1e8c299-bf1a-43d8-bc21-f7dc39124409-kube-api-access-9z4hd\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.641542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.641506    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.641542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.641515    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1e8c299-bf1a-43d8-bc21-f7dc39124409-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.641542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:43.641522    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e8c299-bf1a-43d8-bc21-f7dc39124409-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:35:44.084759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.084716    2583 generic.go:356] "Generic (PLEG): container finished" podID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" containerID="7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f" exitCode=0
Sep 17 19:35:44.085151 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.084761    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"
Sep 17 19:35:44.085151 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.084778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" event={"ID":"f1e8c299-bf1a-43d8-bc21-f7dc39124409","Type":"ContainerDied","Data":"7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f"}
Sep 17 19:35:44.085151 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.084836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" event={"ID":"f1e8c299-bf1a-43d8-bc21-f7dc39124409","Type":"ContainerDied","Data":"82ba73af7652818df9e5ed5c4436bcce8b207c9e25ab11569aa57e1820e2f889"}
Sep 17 19:35:44.085151 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.084859    2583 scope.go:117] "RemoveContainer" containerID="7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f"
Sep 17 19:35:44.094528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.094511    2583 scope.go:117] "RemoveContainer" containerID="7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f"
Sep 17 19:35:44.094812 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:35:44.094796    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f\": container with ID starting with 7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f not found: ID does not exist" containerID="7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f"
Sep 17 19:35:44.094862 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.094821    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f"} err="failed to get container status \"7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f\": rpc error: code = NotFound desc = could not find container \"7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f\": container with ID starting with 7f592ca4b718cd79e02d1907d0574c59180433ee94b174585d5f3d648ac5d77f not found: ID does not exist"
Sep 17 19:35:44.131528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.131502    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"]
Sep 17 19:35:44.140802 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.140778    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v"]
Sep 17 19:35:44.255615 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.255591    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" path="/var/lib/kubelet/pods/f1e8c299-bf1a-43d8-bc21-f7dc39124409/volumes"
Sep 17 19:35:44.358589 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:44.358524    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969pfc5v" podUID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:35:50.309068 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:50.309026    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:35:58.090713 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:35:58.090639    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main" containerID="cri-o://7f1aae564df2e6d3e784a023c954be237c0cb64b039ec66e694ab5a4a49e9b1c" gracePeriod=45
Sep 17 19:36:00.309673 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:00.309640    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:36:10.309066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:10.309028    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:36:10.309440 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:10.309112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:36:20.309316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:20.309236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:36:28.334539 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.334508    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerID="7f1aae564df2e6d3e784a023c954be237c0cb64b039ec66e694ab5a4a49e9b1c" exitCode=0
Sep 17 19:36:28.334910 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.334577    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" event={"ID":"9b239dc0-0db1-4f39-8253-2cc3c58aad57","Type":"ContainerDied","Data":"7f1aae564df2e6d3e784a023c954be237c0cb64b039ec66e694ab5a4a49e9b1c"}
Sep 17 19:36:28.334910 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.334626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf" event={"ID":"9b239dc0-0db1-4f39-8253-2cc3c58aad57","Type":"ContainerDied","Data":"d2d2b18ef7083c495aa1c131b5e6874c50a74547f67fb54d19e81ee96754b674"}
Sep 17 19:36:28.334910 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.334641    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d2b18ef7083c495aa1c131b5e6874c50a74547f67fb54d19e81ee96754b674"
Sep 17 19:36:28.340354 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.340336    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:36:28.391266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.391211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239dc0-0db1-4f39-8253-2cc3c58aad57-tls-certs\") pod \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") "
Sep 17 19:36:28.391369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.391272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49jbm\" (UniqueName: \"kubernetes.io/projected/9b239dc0-0db1-4f39-8253-2cc3c58aad57-kube-api-access-49jbm\") pod \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\" (UID: \"9b239dc0-0db1-4f39-8253-2cc3c58aad57\") "
Sep 17 19:36:28.393349 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.393326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b239dc0-0db1-4f39-8253-2cc3c58aad57-kube-api-access-49jbm" (OuterVolumeSpecName: "kube-api-access-49jbm") pod "9b239dc0-0db1-4f39-8253-2cc3c58aad57" (UID: "9b239dc0-0db1-4f39-8253-2cc3c58aad57"). InnerVolumeSpecName "kube-api-access-49jbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:36:28.393468 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.393397    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b239dc0-0db1-4f39-8253-2cc3c58aad57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b239dc0-0db1-4f39-8253-2cc3c58aad57" (UID: "9b239dc0-0db1-4f39-8253-2cc3c58aad57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:36:28.491978 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.491955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jbm\" (UniqueName: \"kubernetes.io/projected/9b239dc0-0db1-4f39-8253-2cc3c58aad57-kube-api-access-49jbm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:36:28.491978 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:28.491978    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b239dc0-0db1-4f39-8253-2cc3c58aad57-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:36:29.339990 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:29.339959    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"
Sep 17 19:36:29.361937 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:29.361907    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"]
Sep 17 19:36:29.365851 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:29.365826    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebcjjf"]
Sep 17 19:36:30.258328 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:30.258248    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" path="/var/lib/kubelet/pods/9b239dc0-0db1-4f39-8253-2cc3c58aad57/volumes"
Sep 17 19:36:36.082395 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082362    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"]
Sep 17 19:36:36.082847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082749    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b761357b-2604-43a1-8c5f-e3996d8d9481" containerName="main"
Sep 17 19:36:36.082847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082762    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b761357b-2604-43a1-8c5f-e3996d8d9481" containerName="main"
Sep 17 19:36:36.082847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082779    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" containerName="main"
Sep 17 19:36:36.082847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082784    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" containerName="main"
Sep 17 19:36:36.082847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082791    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main"
Sep 17 19:36:36.082847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082796    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main"
Sep 17 19:36:36.083044 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082855    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e8c299-bf1a-43d8-bc21-f7dc39124409" containerName="main"
Sep 17 19:36:36.083044 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082866    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b761357b-2604-43a1-8c5f-e3996d8d9481" containerName="main"
Sep 17 19:36:36.083044 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.082871    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b239dc0-0db1-4f39-8253-2cc3c58aad57" containerName="main"
Sep 17 19:36:36.084961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.084943    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.088093 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.088069    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 19:36:36.088234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.088179    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 19:36:36.088234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.088190    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 19:36:36.088234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.088195    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:36:36.088389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.088197    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 19:36:36.088389 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.088326    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-547w2\""
Sep 17 19:36:36.101528 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.101498    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"]
Sep 17 19:36:36.153964 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.153938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.154069 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.153990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.154069 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.154035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d260e-fa05-401e-ac75-ff0416a55336-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.154181 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.154069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.154181 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.154142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.154181 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.154176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d260e-fa05-401e-ac75-ff0416a55336-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.154348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.154196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.154348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.154217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v429d\" (UniqueName: \"kubernetes.io/projected/0b8d260e-fa05-401e-ac75-ff0416a55336-kube-api-access-v429d\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255502 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d260e-fa05-401e-ac75-ff0416a55336-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255502 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255705 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v429d\" (UniqueName: \"kubernetes.io/projected/0b8d260e-fa05-401e-ac75-ff0416a55336-kube-api-access-v429d\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255705 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255705 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255705 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d260e-fa05-401e-ac75-ff0416a55336-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255924 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255924 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.255924 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.256089 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.255963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.256146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.256099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.256212 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.256195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.256354 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.256333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d260e-fa05-401e-ac75-ff0416a55336-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.258036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.258019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.258431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.258412    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d260e-fa05-401e-ac75-ff0416a55336-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.264036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.264018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v429d\" (UniqueName: \"kubernetes.io/projected/0b8d260e-fa05-401e-ac75-ff0416a55336-kube-api-access-v429d\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.395056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.394981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:36.681390 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:36.681354    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"]
Sep 17 19:36:37.382570 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:37.382534    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" event={"ID":"0b8d260e-fa05-401e-ac75-ff0416a55336","Type":"ContainerStarted","Data":"c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05"}
Sep 17 19:36:37.382993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:37.382593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" event={"ID":"0b8d260e-fa05-401e-ac75-ff0416a55336","Type":"ContainerStarted","Data":"004fba7bad82fd850a81e674a91e20c17616f8d7ceb1cc39cfeee00697d395fb"}
Sep 17 19:36:40.400303 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:40.400259    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerID="c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05" exitCode=0
Sep 17 19:36:40.400707 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:40.400341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" event={"ID":"0b8d260e-fa05-401e-ac75-ff0416a55336","Type":"ContainerDied","Data":"c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05"}
Sep 17 19:36:41.409434 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:41.409400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" event={"ID":"0b8d260e-fa05-401e-ac75-ff0416a55336","Type":"ContainerStarted","Data":"6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c"}
Sep 17 19:36:41.436024 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:41.435966    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podStartSLOduration=2.67908109 podStartE2EDuration="5.435945323s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.756864233s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:41.431984161 +0000 UTC m=+1803.479864019" watchObservedRunningTime="2026-09-17 19:36:41.435945323 +0000 UTC m=+1803.483825181"
Sep 17 19:36:42.395559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:42.395523    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:46.395548 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:46.395511    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:36:46.397335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:46.397304    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 19:36:56.395654 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:36:56.395614    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 19:37:06.395702 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:37:06.395635    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 19:37:16.395433 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:37:16.395388    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 19:37:26.395364 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:37:26.395314    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 19:37:36.395735 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:37:36.395676    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 19:37:46.395860 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:37:46.395814    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Sep 17 19:37:56.407046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:37:56.406970    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:37:56.414901 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:37:56.414877    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:38:20.758983 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:20.758943    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"]
Sep 17 19:38:35.785395 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:35.785335    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" containerID="cri-o://6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c" gracePeriod=45
Sep 17 19:38:36.439997 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.439974    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:38:36.498585 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.498527    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-tmp-dir\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.498585 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.498565    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-dshm\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.498828 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.498596    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d260e-fa05-401e-ac75-ff0416a55336-tls-certs\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.498828 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.498625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v429d\" (UniqueName: \"kubernetes.io/projected/0b8d260e-fa05-401e-ac75-ff0416a55336-kube-api-access-v429d\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.498828 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.498658    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-home\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.499026 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.498999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-kserve-provision-location\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.499154 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.499079    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d260e-fa05-401e-ac75-ff0416a55336-cabundle-cert\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.500012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.499551    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-home" (OuterVolumeSpecName: "home") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:38:36.500012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.499587    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8d260e-fa05-401e-ac75-ff0416a55336-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:38:36.500012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.500001    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-model-cache\") pod \"0b8d260e-fa05-401e-ac75-ff0416a55336\" (UID: \"0b8d260e-fa05-401e-ac75-ff0416a55336\") "
Sep 17 19:38:36.500594 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.500276    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:36.500594 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.500299    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b8d260e-fa05-401e-ac75-ff0416a55336-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:36.500594 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.500556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:38:36.501199 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.501173    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-dshm" (OuterVolumeSpecName: "dshm") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:38:36.501367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.501343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8d260e-fa05-401e-ac75-ff0416a55336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:38:36.501431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.501397    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8d260e-fa05-401e-ac75-ff0416a55336-kube-api-access-v429d" (OuterVolumeSpecName: "kube-api-access-v429d") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "kube-api-access-v429d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:38:36.517210 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.517184    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:38:36.530797 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.530771    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b8d260e-fa05-401e-ac75-ff0416a55336" (UID: "0b8d260e-fa05-401e-ac75-ff0416a55336"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:38:36.600812 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.600789    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:36.600812 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.600810    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:36.600926 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.600819    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:36.600926 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.600827    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d260e-fa05-401e-ac75-ff0416a55336-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:36.600926 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.600835    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v429d\" (UniqueName: \"kubernetes.io/projected/0b8d260e-fa05-401e-ac75-ff0416a55336-kube-api-access-v429d\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:36.600926 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:36.600845    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d260e-fa05-401e-ac75-ff0416a55336-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:38:37.098787 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.098753    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerID="6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c" exitCode=0
Sep 17 19:38:37.099186 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.098793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" event={"ID":"0b8d260e-fa05-401e-ac75-ff0416a55336","Type":"ContainerDied","Data":"6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c"}
Sep 17 19:38:37.099186 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.098804    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"
Sep 17 19:38:37.099186 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.098836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" event={"ID":"0b8d260e-fa05-401e-ac75-ff0416a55336","Type":"ContainerDied","Data":"004fba7bad82fd850a81e674a91e20c17616f8d7ceb1cc39cfeee00697d395fb"}
Sep 17 19:38:37.099186 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.098858    2583 scope.go:117] "RemoveContainer" containerID="6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c"
Sep 17 19:38:37.109081 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.109052    2583 scope.go:117] "RemoveContainer" containerID="c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05"
Sep 17 19:38:37.117718 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.117700    2583 scope.go:117] "RemoveContainer" containerID="6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c"
Sep 17 19:38:37.117986 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:38:37.117965    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c\": container with ID starting with 6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c not found: ID does not exist" containerID="6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c"
Sep 17 19:38:37.118037 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.117994    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c"} err="failed to get container status \"6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c\": rpc error: code = NotFound desc = could not find container \"6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c\": container with ID starting with 6b125f107e68492975de1f13a49fac66132e2fc5814700d1ee7cada7111a581c not found: ID does not exist"
Sep 17 19:38:37.118037 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.118012    2583 scope.go:117] "RemoveContainer" containerID="c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05"
Sep 17 19:38:37.118282 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:38:37.118263    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05\": container with ID starting with c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05 not found: ID does not exist" containerID="c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05"
Sep 17 19:38:37.118325 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.118289    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05"} err="failed to get container status \"c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05\": rpc error: code = NotFound desc = could not find container \"c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05\": container with ID starting with c2d1c9608d3a833896dbe26645870511862f754065db4520311fb1617bccbb05 not found: ID does not exist"
Sep 17 19:38:37.122629 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.122607    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"]
Sep 17 19:38:37.128621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.128598    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc"]
Sep 17 19:38:37.409266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:37.409189    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6766d556c6-rmzhc" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:38:38.254640 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:38:38.254603    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" path="/var/lib/kubelet/pods/0b8d260e-fa05-401e-ac75-ff0416a55336/volumes"
Sep 17 19:39:17.775187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.775154    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"]
Sep 17 19:39:17.775918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.775513    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="storage-initializer"
Sep 17 19:39:17.775918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.775524    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="storage-initializer"
Sep 17 19:39:17.775918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.775543    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main"
Sep 17 19:39:17.775918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.775550    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main"
Sep 17 19:39:17.775918 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.775610    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8d260e-fa05-401e-ac75-ff0416a55336" containerName="main"
Sep 17 19:39:17.786470 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.786444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.789583 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.789553    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 19:39:17.789961 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.789634    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-fmbwz\""
Sep 17 19:39:17.790618 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.790587    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:39:17.790712 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.789635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 19:39:17.790768 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.790741    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 19:39:17.790946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.790922    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 19:39:17.792801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.792777    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"]
Sep 17 19:39:17.904381 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.904511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cjp\" (UniqueName: \"kubernetes.io/projected/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kube-api-access-b5cjp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.904511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.904590 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.904590 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.904659 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c854cb8b-bf91-4cdd-bf26-6c919f23afef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.904659 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:17.904754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:17.904720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.005651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.005817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c854cb8b-bf91-4cdd-bf26-6c919f23afef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.005817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.005817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.005817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.005817 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cjp\" (UniqueName: \"kubernetes.io/projected/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kube-api-access-b5cjp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.006110 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.006110 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.005942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.006224 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.006176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.006281 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.006253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.006625 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.006574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c854cb8b-bf91-4cdd-bf26-6c919f23afef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.006809 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.006703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.006873 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.006840    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.008316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.008292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.008746 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.008673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.014008 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.013989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cjp\" (UniqueName: \"kubernetes.io/projected/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kube-api-access-b5cjp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.098979 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.098956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:18.376375 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:18.376350    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"]
Sep 17 19:39:18.377738 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:39:18.377713    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc854cb8b_bf91_4cdd_bf26_6c919f23afef.slice/crio-76491c59ca6363fca4a0afd2a045638d80eb954b7ec113b89371f3d3e8413462 WatchSource:0}: Error finding container 76491c59ca6363fca4a0afd2a045638d80eb954b7ec113b89371f3d3e8413462: Status 404 returned error can't find the container with id 76491c59ca6363fca4a0afd2a045638d80eb954b7ec113b89371f3d3e8413462
Sep 17 19:39:19.339111 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:19.339079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" event={"ID":"c854cb8b-bf91-4cdd-bf26-6c919f23afef","Type":"ContainerStarted","Data":"a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622"}
Sep 17 19:39:19.339520 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:19.339139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" event={"ID":"c854cb8b-bf91-4cdd-bf26-6c919f23afef","Type":"ContainerStarted","Data":"76491c59ca6363fca4a0afd2a045638d80eb954b7ec113b89371f3d3e8413462"}
Sep 17 19:39:21.352989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:21.352956    2583 generic.go:356] "Generic (PLEG): container finished" podID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerID="a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622" exitCode=0
Sep 17 19:39:21.353286 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:21.353021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" event={"ID":"c854cb8b-bf91-4cdd-bf26-6c919f23afef","Type":"ContainerDied","Data":"a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622"}
Sep 17 19:39:22.361130 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:22.361094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" event={"ID":"c854cb8b-bf91-4cdd-bf26-6c919f23afef","Type":"ContainerStarted","Data":"77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936"}
Sep 17 19:39:22.385544 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:22.385497    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podStartSLOduration=2.567366587 podStartE2EDuration="5.385480755s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.818114168s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:39:22.384481447 +0000 UTC m=+1964.432361305" watchObservedRunningTime="2026-09-17 19:39:22.385480755 +0000 UTC m=+1964.433360615"
Sep 17 19:39:23.099765 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:23.099720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:28.099886 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:28.099840    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:39:28.101724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:28.101671    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:39:38.100284 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:38.100235    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:39:46.585647 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:46.585609    2583 scope.go:117] "RemoveContainer" containerID="7f1aae564df2e6d3e784a023c954be237c0cb64b039ec66e694ab5a4a49e9b1c"
Sep 17 19:39:48.100096 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:48.100049    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:39:58.100333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:39:58.100280    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:40:08.099938 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:08.099891    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:40:18.100072 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:18.100030    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:40:28.099676 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:28.099633    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:40:38.109623 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:38.109592    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:40:38.117280 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:38.117255    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:40:39.582030 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:39.581996    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"]
Sep 17 19:40:54.846491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:54.846403    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" containerID="cri-o://77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936" gracePeriod=45
Sep 17 19:40:55.110573 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.110480    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 17 19:40:55.397418 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.397387    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:40:55.536364 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c854cb8b-bf91-4cdd-bf26-6c919f23afef-cabundle-cert\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536381    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-dshm\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-model-cache\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5cjp\" (UniqueName: \"kubernetes.io/projected/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kube-api-access-b5cjp\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536479    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tls-certs\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536497    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-home\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536523    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tmp-dir\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536548    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kserve-provision-location\") pod \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\" (UID: \"c854cb8b-bf91-4cdd-bf26-6c919f23afef\") "
Sep 17 19:40:55.536941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.536811    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-model-cache" (OuterVolumeSpecName: "model-cache") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:55.537070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.537043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-home" (OuterVolumeSpecName: "home") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:55.537168 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.537083    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:55.537438 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.537415    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c854cb8b-bf91-4cdd-bf26-6c919f23afef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:40:55.538890 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.538868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-dshm" (OuterVolumeSpecName: "dshm") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:55.539435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.539413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kube-api-access-b5cjp" (OuterVolumeSpecName: "kube-api-access-b5cjp") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "kube-api-access-b5cjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:40:55.539529 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.539417    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:40:55.561508 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.561485    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c854cb8b-bf91-4cdd-bf26-6c919f23afef" (UID: "c854cb8b-bf91-4cdd-bf26-6c919f23afef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637705    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5cjp\" (UniqueName: \"kubernetes.io/projected/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kube-api-access-b5cjp\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637741    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637750    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637758    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637766    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637775    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c854cb8b-bf91-4cdd-bf26-6c919f23afef-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637783    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.637798 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.637790    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854cb8b-bf91-4cdd-bf26-6c919f23afef-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:40:55.914894 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.914799    2583 generic.go:356] "Generic (PLEG): container finished" podID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerID="77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936" exitCode=0
Sep 17 19:40:55.914894 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.914853    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"
Sep 17 19:40:55.914894 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.914879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" event={"ID":"c854cb8b-bf91-4cdd-bf26-6c919f23afef","Type":"ContainerDied","Data":"77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936"}
Sep 17 19:40:55.915454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.914931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89" event={"ID":"c854cb8b-bf91-4cdd-bf26-6c919f23afef","Type":"ContainerDied","Data":"76491c59ca6363fca4a0afd2a045638d80eb954b7ec113b89371f3d3e8413462"}
Sep 17 19:40:55.915454 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.914954    2583 scope.go:117] "RemoveContainer" containerID="77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936"
Sep 17 19:40:55.925140 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.925123    2583 scope.go:117] "RemoveContainer" containerID="a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622"
Sep 17 19:40:55.933378 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.933361    2583 scope.go:117] "RemoveContainer" containerID="77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936"
Sep 17 19:40:55.933632 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:40:55.933616    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936\": container with ID starting with 77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936 not found: ID does not exist" containerID="77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936"
Sep 17 19:40:55.933674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.933639    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936"} err="failed to get container status \"77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936\": rpc error: code = NotFound desc = could not find container \"77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936\": container with ID starting with 77ca663b9a43a283d1a703fbb58d82fc352f07e432ea2f06798c6f314aa06936 not found: ID does not exist"
Sep 17 19:40:55.933674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.933653    2583 scope.go:117] "RemoveContainer" containerID="a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622"
Sep 17 19:40:55.933899 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:40:55.933884    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622\": container with ID starting with a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622 not found: ID does not exist" containerID="a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622"
Sep 17 19:40:55.933939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.933902    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622"} err="failed to get container status \"a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622\": rpc error: code = NotFound desc = could not find container \"a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622\": container with ID starting with a3507b93f9323f9be80e8fd60218532bd64957e1b5d3621b868a171d4d7c5622 not found: ID does not exist"
Sep 17 19:40:55.939377 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.939357    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"]
Sep 17 19:40:55.954185 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:55.954164    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5dccf6b755-thp89"]
Sep 17 19:40:56.255974 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:40:56.255903    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" path="/var/lib/kubelet/pods/c854cb8b-bf91-4cdd-bf26-6c919f23afef/volumes"
Sep 17 19:41:33.777371 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.777333    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"]
Sep 17 19:41:33.777795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.777710    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="storage-initializer"
Sep 17 19:41:33.777795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.777723    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="storage-initializer"
Sep 17 19:41:33.777795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.777748    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main"
Sep 17 19:41:33.777795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.777753    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main"
Sep 17 19:41:33.777936 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.777823    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c854cb8b-bf91-4cdd-bf26-6c919f23afef" containerName="main"
Sep 17 19:41:33.780889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.780872    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.783693 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.783664    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-5zt5s\""
Sep 17 19:41:33.783863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.783736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 19:41:33.783863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.783753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:41:33.783863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.783755    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 19:41:33.783863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.783675    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 19:41:33.783863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.783675    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 19:41:33.791271 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.791247    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"]
Sep 17 19:41:33.803361 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-home\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.803475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlv5\" (UniqueName: \"kubernetes.io/projected/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kube-api-access-2mlv5\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.803475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-dshm\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.803475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803443    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kserve-provision-location\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.803584 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tmp-dir\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.803653 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-model-cache\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.803725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tls-certs\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.803769 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.803755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86b19af0-a979-4975-9c93-b5ff1d6f9bab-cabundle-cert\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.904995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.904965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86b19af0-a979-4975-9c93-b5ff1d6f9bab-cabundle-cert\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905157 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-home\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905157 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlv5\" (UniqueName: \"kubernetes.io/projected/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kube-api-access-2mlv5\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-dshm\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905268 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kserve-provision-location\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tmp-dir\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-model-cache\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905486 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tls-certs\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-home\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kserve-provision-location\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905750 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86b19af0-a979-4975-9c93-b5ff1d6f9bab-cabundle-cert\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905837 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tmp-dir\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.905837 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.905829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-model-cache\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.907743 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.907721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-dshm\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.908008 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.907992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tls-certs\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:33.914417 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:33.914390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlv5\" (UniqueName: \"kubernetes.io/projected/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kube-api-access-2mlv5\") pod \"stop-feature-test-kserve-699c6fc5bf-dqxv2\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:34.092148 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:34.092118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:34.373443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:34.373415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"]
Sep 17 19:41:34.375095 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:41:34.375060    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b19af0_a979_4975_9c93_b5ff1d6f9bab.slice/crio-1e3687a7cea7b1e1a83d058f5e30fc39c99c15d093b4959edcdb484fefea6e62 WatchSource:0}: Error finding container 1e3687a7cea7b1e1a83d058f5e30fc39c99c15d093b4959edcdb484fefea6e62: Status 404 returned error can't find the container with id 1e3687a7cea7b1e1a83d058f5e30fc39c99c15d093b4959edcdb484fefea6e62
Sep 17 19:41:35.148145 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:35.148104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" event={"ID":"86b19af0-a979-4975-9c93-b5ff1d6f9bab","Type":"ContainerStarted","Data":"565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220"}
Sep 17 19:41:35.148518 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:35.148189    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" event={"ID":"86b19af0-a979-4975-9c93-b5ff1d6f9bab","Type":"ContainerStarted","Data":"1e3687a7cea7b1e1a83d058f5e30fc39c99c15d093b4959edcdb484fefea6e62"}
Sep 17 19:41:37.160571 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:37.160540    2583 generic.go:356] "Generic (PLEG): container finished" podID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerID="565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220" exitCode=0
Sep 17 19:41:37.160981 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:37.160620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" event={"ID":"86b19af0-a979-4975-9c93-b5ff1d6f9bab","Type":"ContainerDied","Data":"565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220"}
Sep 17 19:41:38.168187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:38.168145    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" event={"ID":"86b19af0-a979-4975-9c93-b5ff1d6f9bab","Type":"ContainerStarted","Data":"caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b"}
Sep 17 19:41:38.194487 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:38.194430    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podStartSLOduration=2.8465549340000003 podStartE2EDuration="5.194414886s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.347859952s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:41:38.190783789 +0000 UTC m=+2100.238663643" watchObservedRunningTime="2026-09-17 19:41:38.194414886 +0000 UTC m=+2100.242294742"
Sep 17 19:41:39.092965 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:39.092930    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:44.092752 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:44.092717    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:41:44.094194 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:44.094166    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:41:54.092913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:41:54.092867    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:42:04.093270 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:04.093221    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:42:14.092777 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:14.092730    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:42:24.092821 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:24.092719    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:42:34.092739 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:34.092661    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:42:44.092950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:44.092911    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:42:54.092967 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:54.092925    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:42:55.555934 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.555896    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"]
Sep 17 19:42:55.560414 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.560389    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.563046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.563022    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 19:42:55.563265 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.563242    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:42:55.563265 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.563116    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 19:42:55.563448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.563335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 19:42:55.563448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.563194    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 19:42:55.563533 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.563454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-nk9qh\""
Sep 17 19:42:55.574680 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.574653    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"]
Sep 17 19:42:55.706491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6tg6\" (UniqueName: \"kubernetes.io/projected/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kube-api-access-p6tg6\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.706491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-model-cache\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.706810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.706810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tls-certs\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.706810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-dshm\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.706810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.706810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tmp-dir\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.707023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.706845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-home\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-home\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tg6\" (UniqueName: \"kubernetes.io/projected/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kube-api-access-p6tg6\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-model-cache\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tls-certs\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-dshm\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.807889 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.807817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tmp-dir\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.808218 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.808038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-home\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.808280 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.808224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tmp-dir\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.808508 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.808489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.808946 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.808897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-cabundle-cert\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.809386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.809359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-model-cache\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.811048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.811015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tls-certs\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.811156 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.811097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-dshm\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.816750 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.816726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tg6\" (UniqueName: \"kubernetes.io/projected/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kube-api-access-p6tg6\") pod \"custom-route-timeout-test-kserve-68b977ddc5-4bgwg\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:55.872828 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:55.872796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:42:56.155581 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:56.155545    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"]
Sep 17 19:42:56.155957 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:42:56.155929    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f0ae66_9b5c_4ca1_a6b5_cdd571e93ed7.slice/crio-94f0630c606b42f59ea42a316613f5b91b56099b643f30a79213bbb139a01a80 WatchSource:0}: Error finding container 94f0630c606b42f59ea42a316613f5b91b56099b643f30a79213bbb139a01a80: Status 404 returned error can't find the container with id 94f0630c606b42f59ea42a316613f5b91b56099b643f30a79213bbb139a01a80
Sep 17 19:42:56.616782 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:56.616744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" event={"ID":"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7","Type":"ContainerStarted","Data":"05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c"}
Sep 17 19:42:56.617242 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:56.616824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" event={"ID":"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7","Type":"ContainerStarted","Data":"94f0630c606b42f59ea42a316613f5b91b56099b643f30a79213bbb139a01a80"}
Sep 17 19:42:58.632420 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:58.632391    2583 generic.go:356] "Generic (PLEG): container finished" podID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerID="05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c" exitCode=0
Sep 17 19:42:58.632760 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:58.632447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" event={"ID":"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7","Type":"ContainerDied","Data":"05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c"}
Sep 17 19:42:59.641048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:59.641008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" event={"ID":"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7","Type":"ContainerStarted","Data":"9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361"}
Sep 17 19:42:59.665031 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:59.664965    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podStartSLOduration=2.304295745 podStartE2EDuration="4.66494384s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.360648095s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:42:59.661987816 +0000 UTC m=+2181.709867685" watchObservedRunningTime="2026-09-17 19:42:59.66494384 +0000 UTC m=+2181.712823699"
Sep 17 19:42:59.873518 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:42:59.873484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:43:04.093001 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:04.092946    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:43:05.873846 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:05.873810    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:43:05.875229 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:05.875195    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:43:14.093670 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:14.093620    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:43:15.873387 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:15.873343    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:43:24.093164 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:24.093119    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:43:25.873480 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:25.873434    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:43:34.108731 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:34.108702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:43:34.120835 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:34.120811    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:43:35.874070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:35.874024    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:43:35.932754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:35.932719    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"]
Sep 17 19:43:45.873612 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:45.873565    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:43:50.958075 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:50.957967    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" containerID="cri-o://caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b" gracePeriod=45
Sep 17 19:43:51.109985 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.109948    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 17 19:43:51.713224 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.713195    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:43:51.829123 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tmp-dir\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829273 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829144    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mlv5\" (UniqueName: \"kubernetes.io/projected/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kube-api-access-2mlv5\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829273 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829166    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tls-certs\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829273 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829192    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kserve-provision-location\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829273 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-model-cache\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829273 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829270    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-home\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829531 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-dshm\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829531 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86b19af0-a979-4975-9c93-b5ff1d6f9bab-cabundle-cert\") pod \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\" (UID: \"86b19af0-a979-4975-9c93-b5ff1d6f9bab\") "
Sep 17 19:43:51.829642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829544    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-model-cache" (OuterVolumeSpecName: "model-cache") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:51.829716 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.829661    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.830036 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.830007    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-home" (OuterVolumeSpecName: "home") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:51.830205 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.830068    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b19af0-a979-4975-9c93-b5ff1d6f9bab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:43:51.832160 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.832121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-dshm" (OuterVolumeSpecName: "dshm") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:51.832272 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.832206    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:43:51.832320 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.832302    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kube-api-access-2mlv5" (OuterVolumeSpecName: "kube-api-access-2mlv5") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "kube-api-access-2mlv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:43:51.847735 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.847709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:51.867501 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.867465    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86b19af0-a979-4975-9c93-b5ff1d6f9bab" (UID: "86b19af0-a979-4975-9c93-b5ff1d6f9bab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:43:51.930863 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.930834    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.931009 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.930869    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.931009 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.930884    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.931009 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.930897    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86b19af0-a979-4975-9c93-b5ff1d6f9bab-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.931009 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.930912    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.931009 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.930924    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlv5\" (UniqueName: \"kubernetes.io/projected/86b19af0-a979-4975-9c93-b5ff1d6f9bab-kube-api-access-2mlv5\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.931009 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.930935    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86b19af0-a979-4975-9c93-b5ff1d6f9bab-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:43:51.955309 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.955277    2583 generic.go:356] "Generic (PLEG): container finished" podID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerID="caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b" exitCode=0
Sep 17 19:43:51.955406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.955321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" event={"ID":"86b19af0-a979-4975-9c93-b5ff1d6f9bab","Type":"ContainerDied","Data":"caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b"}
Sep 17 19:43:51.955406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.955348    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"
Sep 17 19:43:51.955406 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.955388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2" event={"ID":"86b19af0-a979-4975-9c93-b5ff1d6f9bab","Type":"ContainerDied","Data":"1e3687a7cea7b1e1a83d058f5e30fc39c99c15d093b4959edcdb484fefea6e62"}
Sep 17 19:43:51.955527 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.955410    2583 scope.go:117] "RemoveContainer" containerID="caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b"
Sep 17 19:43:51.964819 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.964606    2583 scope.go:117] "RemoveContainer" containerID="565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220"
Sep 17 19:43:51.974090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.974071    2583 scope.go:117] "RemoveContainer" containerID="caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b"
Sep 17 19:43:51.974375 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:43:51.974352    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b\": container with ID starting with caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b not found: ID does not exist" containerID="caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b"
Sep 17 19:43:51.974507 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.974382    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b"} err="failed to get container status \"caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b\": rpc error: code = NotFound desc = could not find container \"caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b\": container with ID starting with caf7cf1b55e6db82fd61e19cce819198399b2ee0974cdb875921b38c2d16e60b not found: ID does not exist"
Sep 17 19:43:51.974507 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.974399    2583 scope.go:117] "RemoveContainer" containerID="565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220"
Sep 17 19:43:51.974719 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:43:51.974697    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220\": container with ID starting with 565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220 not found: ID does not exist" containerID="565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220"
Sep 17 19:43:51.974771 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.974726    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220"} err="failed to get container status \"565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220\": rpc error: code = NotFound desc = could not find container \"565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220\": container with ID starting with 565b3463c82ba14ef2d25494211e8f1dae91d98568cdf29d62c30009bcef8220 not found: ID does not exist"
Sep 17 19:43:51.986952 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.986922    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"]
Sep 17 19:43:51.990565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:51.990542    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-dqxv2"]
Sep 17 19:43:52.257285 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:52.257216    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" path="/var/lib/kubelet/pods/86b19af0-a979-4975-9c93-b5ff1d6f9bab/volumes"
Sep 17 19:43:55.873759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:43:55.873711    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:44:05.873892 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:05.873840    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:44:15.874229 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:15.874185    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:44:19.648986 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.648948    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"]
Sep 17 19:44:19.649562 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.649545    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="storage-initializer"
Sep 17 19:44:19.649610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.649566    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="storage-initializer"
Sep 17 19:44:19.649610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.649578    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main"
Sep 17 19:44:19.649610 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.649586    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main"
Sep 17 19:44:19.649737 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.649712    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86b19af0-a979-4975-9c93-b5ff1d6f9bab" containerName="main"
Sep 17 19:44:19.653222 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.653201    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.656017 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.655992    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 19:44:19.656181 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.656093    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 19:44:19.656261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.656184    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-5zt5s\""
Sep 17 19:44:19.656261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.656091    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:44:19.656538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.656500    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 19:44:19.656657 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.656510    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 19:44:19.665565 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.665543    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"]
Sep 17 19:44:19.765655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-home\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.765826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-model-cache\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.765826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-kserve-provision-location\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.765826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-dshm\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.765826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd520f-95ee-4774-8251-6236728f9c3c-cabundle-cert\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.765995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrm7\" (UniqueName: \"kubernetes.io/projected/dbfd520f-95ee-4774-8251-6236728f9c3c-kube-api-access-fvrm7\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.765995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-tmp-dir\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.765995 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.765911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd520f-95ee-4774-8251-6236728f9c3c-tls-certs\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867032 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.866990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-kserve-provision-location\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867203 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-dshm\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867203 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd520f-95ee-4774-8251-6236728f9c3c-cabundle-cert\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867203 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrm7\" (UniqueName: \"kubernetes.io/projected/dbfd520f-95ee-4774-8251-6236728f9c3c-kube-api-access-fvrm7\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867390 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-tmp-dir\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867390 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd520f-95ee-4774-8251-6236728f9c3c-tls-certs\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867390 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-home\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867390 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-model-cache\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867609 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-kserve-provision-location\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-tmp-dir\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-home\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.867974 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.867947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-model-cache\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.868111 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.868089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd520f-95ee-4774-8251-6236728f9c3c-cabundle-cert\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.869711 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.869671    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-dshm\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.870105 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.870078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd520f-95ee-4774-8251-6236728f9c3c-tls-certs\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.876187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.876167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrm7\" (UniqueName: \"kubernetes.io/projected/dbfd520f-95ee-4774-8251-6236728f9c3c-kube-api-access-fvrm7\") pod \"stop-feature-test-kserve-699c6fc5bf-5w7l9\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:19.967414 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:19.967343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:20.253854 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:44:20.253822    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfd520f_95ee_4774_8251_6236728f9c3c.slice/crio-d1d3a96677b5e1b4baa99a46582f08a3fb956027bdafb944034bb3afd2ecdc16 WatchSource:0}: Error finding container d1d3a96677b5e1b4baa99a46582f08a3fb956027bdafb944034bb3afd2ecdc16: Status 404 returned error can't find the container with id d1d3a96677b5e1b4baa99a46582f08a3fb956027bdafb944034bb3afd2ecdc16
Sep 17 19:44:20.263220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:20.263189    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"]
Sep 17 19:44:21.120384 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:21.120341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" event={"ID":"dbfd520f-95ee-4774-8251-6236728f9c3c","Type":"ContainerStarted","Data":"e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb"}
Sep 17 19:44:21.120810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:21.120415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" event={"ID":"dbfd520f-95ee-4774-8251-6236728f9c3c","Type":"ContainerStarted","Data":"d1d3a96677b5e1b4baa99a46582f08a3fb956027bdafb944034bb3afd2ecdc16"}
Sep 17 19:44:23.139756 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:23.139722    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerID="e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb" exitCode=0
Sep 17 19:44:23.140134 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:23.139763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" event={"ID":"dbfd520f-95ee-4774-8251-6236728f9c3c","Type":"ContainerDied","Data":"e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb"}
Sep 17 19:44:24.151101 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:24.151066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" event={"ID":"dbfd520f-95ee-4774-8251-6236728f9c3c","Type":"ContainerStarted","Data":"f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e"}
Sep 17 19:44:24.200065 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:24.200010    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podStartSLOduration=2.860591523 podStartE2EDuration="5.199993423s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.3394019s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:44:24.197838349 +0000 UTC m=+2266.245718209" watchObservedRunningTime="2026-09-17 19:44:24.199993423 +0000 UTC m=+2266.247873279"
Sep 17 19:44:24.967603 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:24.967567    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:25.873550 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:25.873506    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:44:29.967523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:29.967481    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:44:29.969386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:29.969352    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:44:35.873950 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:35.873902    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:44:39.968345 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:39.968291    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:44:45.874146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:45.874088    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:44:49.967971 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:49.967924    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:44:55.884248 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:55.884213    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:44:55.892500 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:55.892470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:44:59.968794 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:44:59.968747    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:45:00.424931 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:00.424895    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"]
Sep 17 19:45:09.968302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:09.968245    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:45:15.447815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:15.447741    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" containerID="cri-o://9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361" gracePeriod=45
Sep 17 19:45:15.885003 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:15.884958    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 17 19:45:16.202170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.202144    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:45:16.259256 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259224    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kserve-provision-location\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.259457 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259328    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tls-certs\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.259457 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6tg6\" (UniqueName: \"kubernetes.io/projected/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kube-api-access-p6tg6\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.259457 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-dshm\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.259457 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259426    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-cabundle-cert\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.259707 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259465    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-model-cache\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.259707 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-home\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.259707 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259575    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tmp-dir\") pod \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\" (UID: \"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7\") "
Sep 17 19:45:16.260832 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.259900    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-model-cache" (OuterVolumeSpecName: "model-cache") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:16.260832 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.260290    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:45:16.260832 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.260788    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-home" (OuterVolumeSpecName: "home") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:16.262135 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.262103    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kube-api-access-p6tg6" (OuterVolumeSpecName: "kube-api-access-p6tg6") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "kube-api-access-p6tg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:45:16.262239 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.262149    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:45:16.262239 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.262154    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-dshm" (OuterVolumeSpecName: "dshm") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:16.280919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.280890    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:16.287547 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.287524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" (UID: "72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:45:16.361005 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.360978    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.361005 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.361001    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.361142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.361011    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.361142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.361021    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6tg6\" (UniqueName: \"kubernetes.io/projected/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-kube-api-access-p6tg6\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.361142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.361030    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.361142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.361038    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.361142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.361045    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.361142 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.361052    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:45:16.466381 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.466288    2583 generic.go:356] "Generic (PLEG): container finished" podID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerID="9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361" exitCode=0
Sep 17 19:45:16.466381 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.466344    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"
Sep 17 19:45:16.466381 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.466370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" event={"ID":"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7","Type":"ContainerDied","Data":"9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361"}
Sep 17 19:45:16.466988 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.466426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg" event={"ID":"72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7","Type":"ContainerDied","Data":"94f0630c606b42f59ea42a316613f5b91b56099b643f30a79213bbb139a01a80"}
Sep 17 19:45:16.466988 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.466448    2583 scope.go:117] "RemoveContainer" containerID="9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361"
Sep 17 19:45:16.479164 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.478812    2583 scope.go:117] "RemoveContainer" containerID="05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c"
Sep 17 19:45:16.487473 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.487453    2583 scope.go:117] "RemoveContainer" containerID="9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361"
Sep 17 19:45:16.487837 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:45:16.487786    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361\": container with ID starting with 9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361 not found: ID does not exist" containerID="9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361"
Sep 17 19:45:16.488001 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.487816    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361"} err="failed to get container status \"9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361\": rpc error: code = NotFound desc = could not find container \"9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361\": container with ID starting with 9340bab9bea6714ffa21d0f1c57ca1cdf2ed9bf31ae033baeb2b9c79011ae361 not found: ID does not exist"
Sep 17 19:45:16.488001 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.487874    2583 scope.go:117] "RemoveContainer" containerID="05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c"
Sep 17 19:45:16.488316 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:45:16.488253    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c\": container with ID starting with 05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c not found: ID does not exist" containerID="05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c"
Sep 17 19:45:16.488316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.488295    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c"} err="failed to get container status \"05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c\": rpc error: code = NotFound desc = could not find container \"05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c\": container with ID starting with 05a163bb9c8057f835d6a1df375407232b4e343532180aea05a07ed0ca00cd2c not found: ID does not exist"
Sep 17 19:45:16.491848 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.491813    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"]
Sep 17 19:45:16.494296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:16.494276    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-68b977ddc5-4bgwg"]
Sep 17 19:45:18.256491 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:18.256451    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" path="/var/lib/kubelet/pods/72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7/volumes"
Sep 17 19:45:19.968378 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:19.968290    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:45:29.968172 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:29.968128    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:45:39.968261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:39.968216    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:45:49.968105 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:49.968053    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:45:59.968825 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:45:59.968765    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:46:09.967878 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:09.967831    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:46:19.977649 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:19.977611    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:46:19.985420 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:19.985391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:46:23.685128 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:23.685091    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"]
Sep 17 19:46:38.711048 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:38.710983    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" containerID="cri-o://f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e" gracePeriod=45
Sep 17 19:46:38.979118 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:38.979031    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Sep 17 19:46:39.462372 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.462348    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:46:39.568275 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-home\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.568432 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd520f-95ee-4774-8251-6236728f9c3c-tls-certs\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.568432 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568321    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvrm7\" (UniqueName: \"kubernetes.io/projected/dbfd520f-95ee-4774-8251-6236728f9c3c-kube-api-access-fvrm7\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.568432 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568366    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-kserve-provision-location\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.568432 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568392    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-tmp-dir\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.568432 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-model-cache\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.568730 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568472    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-dshm\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.568801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568722    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:39.568970 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.568950    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd520f-95ee-4774-8251-6236728f9c3c-cabundle-cert\") pod \"dbfd520f-95ee-4774-8251-6236728f9c3c\" (UID: \"dbfd520f-95ee-4774-8251-6236728f9c3c\") "
Sep 17 19:46:39.569321 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.569261    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-home" (OuterVolumeSpecName: "home") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:39.569440 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.569332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfd520f-95ee-4774-8251-6236728f9c3c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:46:39.569556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.569533    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfd520f-95ee-4774-8251-6236728f9c3c-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.569622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.569562    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.569622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.569577    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.570842 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.570819    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfd520f-95ee-4774-8251-6236728f9c3c-kube-api-access-fvrm7" (OuterVolumeSpecName: "kube-api-access-fvrm7") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "kube-api-access-fvrm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:46:39.571064 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.571033    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-dshm" (OuterVolumeSpecName: "dshm") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:39.571171 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.571065    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfd520f-95ee-4774-8251-6236728f9c3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:46:39.582161 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.582124    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:39.591417 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.591391    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbfd520f-95ee-4774-8251-6236728f9c3c" (UID: "dbfd520f-95ee-4774-8251-6236728f9c3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:39.669904 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.669883    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbfd520f-95ee-4774-8251-6236728f9c3c-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.669904 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.669903    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvrm7\" (UniqueName: \"kubernetes.io/projected/dbfd520f-95ee-4774-8251-6236728f9c3c-kube-api-access-fvrm7\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.670023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.669914    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.670023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.669922    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.670023 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.669932    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbfd520f-95ee-4774-8251-6236728f9c3c-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:46:39.955813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.955730    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerID="f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e" exitCode=0
Sep 17 19:46:39.955813 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.955793    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"
Sep 17 19:46:39.956278 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.955815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" event={"ID":"dbfd520f-95ee-4774-8251-6236728f9c3c","Type":"ContainerDied","Data":"f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e"}
Sep 17 19:46:39.956278 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.955871    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9" event={"ID":"dbfd520f-95ee-4774-8251-6236728f9c3c","Type":"ContainerDied","Data":"d1d3a96677b5e1b4baa99a46582f08a3fb956027bdafb944034bb3afd2ecdc16"}
Sep 17 19:46:39.956278 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.955896    2583 scope.go:117] "RemoveContainer" containerID="f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e"
Sep 17 19:46:39.966919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.966895    2583 scope.go:117] "RemoveContainer" containerID="e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb"
Sep 17 19:46:39.976226 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.976209    2583 scope.go:117] "RemoveContainer" containerID="f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e"
Sep 17 19:46:39.976530 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:46:39.976503    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e\": container with ID starting with f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e not found: ID does not exist" containerID="f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e"
Sep 17 19:46:39.976624 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.976537    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e"} err="failed to get container status \"f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e\": rpc error: code = NotFound desc = could not find container \"f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e\": container with ID starting with f20ce167790eacae9ba2c18832c44e07de50fea020f4e253fd5b4e718777a29e not found: ID does not exist"
Sep 17 19:46:39.976624 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.976558    2583 scope.go:117] "RemoveContainer" containerID="e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb"
Sep 17 19:46:39.976873 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:46:39.976852    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb\": container with ID starting with e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb not found: ID does not exist" containerID="e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb"
Sep 17 19:46:39.976920 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.976881    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb"} err="failed to get container status \"e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb\": rpc error: code = NotFound desc = could not find container \"e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb\": container with ID starting with e56e694083622deb1a125b26c5521af01e2a04c70c618ed456322efb588e4dcb not found: ID does not exist"
Sep 17 19:46:39.980647 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.980621    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"]
Sep 17 19:46:39.986115 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:39.986078    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-699c6fc5bf-5w7l9"]
Sep 17 19:46:40.255287 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:46:40.255214    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" path="/var/lib/kubelet/pods/dbfd520f-95ee-4774-8251-6236728f9c3c/volumes"
Sep 17 19:47:22.968907 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.968872    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"]
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969238    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="storage-initializer"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969249    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="storage-initializer"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969257    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="storage-initializer"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969263    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="storage-initializer"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969274    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969280    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969286    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969291    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969346    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f0ae66-9b5c-4ca1-a6b5-cdd571e93ed7" containerName="main"
Sep 17 19:47:22.969393 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.969355    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbfd520f-95ee-4774-8251-6236728f9c3c" containerName="main"
Sep 17 19:47:22.972302 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.972252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:22.975056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.975032    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-8zvrx\""
Sep 17 19:47:22.975260 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.975077    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:47:22.975396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.975259    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 19:47:22.975396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.975214    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 19:47:22.975396 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.975229    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 19:47:22.975568 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.975421    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 19:47:22.987034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:22.987013    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"]
Sep 17 19:47:23.107672 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-tmp-dir\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.107823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423ccc2a-118e-4382-9c6a-62830331e5a1-tls-certs\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.107823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-dshm\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.107823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.107823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ccc2a-118e-4382-9c6a-62830331e5a1-cabundle-cert\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.107823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwll\" (UniqueName: \"kubernetes.io/projected/423ccc2a-118e-4382-9c6a-62830331e5a1-kube-api-access-wzwll\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.108035 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-model-cache\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.108035 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.107914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-home\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.208948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.208919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423ccc2a-118e-4382-9c6a-62830331e5a1-tls-certs\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.208948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.208949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-dshm\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.208971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ccc2a-118e-4382-9c6a-62830331e5a1-cabundle-cert\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwll\" (UniqueName: \"kubernetes.io/projected/423ccc2a-118e-4382-9c6a-62830331e5a1-kube-api-access-wzwll\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-model-cache\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209316 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-home\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209429 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-tmp-dir\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209486 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209463    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209656 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-model-cache\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209764 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-tmp-dir\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.209764 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.209722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-home\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.210063 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.210039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ccc2a-118e-4382-9c6a-62830331e5a1-cabundle-cert\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.211441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.211423    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-dshm\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.211729 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.211710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423ccc2a-118e-4382-9c6a-62830331e5a1-tls-certs\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.220051 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.219999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwll\" (UniqueName: \"kubernetes.io/projected/423ccc2a-118e-4382-9c6a-62830331e5a1-kube-api-access-wzwll\") pod \"lora-single-adapter-test-kserve-fccb4d955-sw9k2\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.283513 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.283492    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:23.568046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:23.568016    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"]
Sep 17 19:47:23.569079 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:47:23.569045    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423ccc2a_118e_4382_9c6a_62830331e5a1.slice/crio-b1f770c01f81a2dcd5440362e07c2a9524220e57fc4cb6eb28288b0f3e18b442 WatchSource:0}: Error finding container b1f770c01f81a2dcd5440362e07c2a9524220e57fc4cb6eb28288b0f3e18b442: Status 404 returned error can't find the container with id b1f770c01f81a2dcd5440362e07c2a9524220e57fc4cb6eb28288b0f3e18b442
Sep 17 19:47:24.215584 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:24.215538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" event={"ID":"423ccc2a-118e-4382-9c6a-62830331e5a1","Type":"ContainerStarted","Data":"6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237"}
Sep 17 19:47:24.215994 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:24.215613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" event={"ID":"423ccc2a-118e-4382-9c6a-62830331e5a1","Type":"ContainerStarted","Data":"b1f770c01f81a2dcd5440362e07c2a9524220e57fc4cb6eb28288b0f3e18b442"}
Sep 17 19:47:27.235383 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:27.235342    2583 generic.go:356] "Generic (PLEG): container finished" podID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerID="6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237" exitCode=0
Sep 17 19:47:27.235754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:27.235416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" event={"ID":"423ccc2a-118e-4382-9c6a-62830331e5a1","Type":"ContainerDied","Data":"6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237"}
Sep 17 19:47:28.242789 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:28.242741    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" event={"ID":"423ccc2a-118e-4382-9c6a-62830331e5a1","Type":"ContainerStarted","Data":"afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046"}
Sep 17 19:47:28.268922 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:28.268875    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podStartSLOduration=3.51577685 podStartE2EDuration="6.268859952s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.753083102s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:47:28.265421598 +0000 UTC m=+2450.313301454" watchObservedRunningTime="2026-09-17 19:47:28.268859952 +0000 UTC m=+2450.316739808"
Sep 17 19:47:28.283755 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:28.283719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:33.284630 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:33.284588    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:47:33.286264 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:33.286234    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:47:43.284513 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:43.284474    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:47:53.284846 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:47:53.284803    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:48:03.284574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:48:03.284526    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:48:13.284049 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:48:13.284003    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:48:23.284054 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:48:23.283967    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:48:33.284852 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:48:33.284812    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:48:43.294356 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:48:43.294322    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:48:43.307776 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:48:43.307747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:49:07.313917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:07.313887    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-fccb4d955-sw9k2_423ccc2a-118e-4382-9c6a-62830331e5a1/storage-initializer/0.log"
Sep 17 19:49:07.323477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:07.323456    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-fccb4d955-sw9k2_423ccc2a-118e-4382-9c6a-62830331e5a1/main/0.log"
Sep 17 19:49:12.535803 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.535728    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"]
Sep 17 19:49:12.540841 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.540818    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.544448 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.544426    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-w655m\""
Sep 17 19:49:12.544549 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.544480    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 19:49:12.545020 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.544995    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:49:12.545321 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.545297    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 19:49:12.545441 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.545420    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 19:49:12.545513 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.545436    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 19:49:12.557764 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.557740    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"]
Sep 17 19:49:12.647427 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.647427 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.647600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.647600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbde963-daf1-456f-b2e0-c712ab431aec-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.647600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbde963-daf1-456f-b2e0-c712ab431aec-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.647600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbsfm\" (UniqueName: \"kubernetes.io/projected/0bbde963-daf1-456f-b2e0-c712ab431aec-kube-api-access-tbsfm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.647600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.647600 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.647577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748068 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748187 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbde963-daf1-456f-b2e0-c712ab431aec-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbde963-daf1-456f-b2e0-c712ab431aec-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbsfm\" (UniqueName: \"kubernetes.io/projected/0bbde963-daf1-456f-b2e0-c712ab431aec-kube-api-access-tbsfm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748532 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748591 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748842 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.748956 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.748896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.749335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.749314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbde963-daf1-456f-b2e0-c712ab431aec-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.750660 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.750638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.750959 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.750942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbde963-daf1-456f-b2e0-c712ab431aec-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.756717 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.756678    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbsfm\" (UniqueName: \"kubernetes.io/projected/0bbde963-daf1-456f-b2e0-c712ab431aec-kube-api-access-tbsfm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:12.857847 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:12.857809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:13.142895 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:13.142806    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"]
Sep 17 19:49:13.145824 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:49:13.145794    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbde963_daf1_456f_b2e0_c712ab431aec.slice/crio-044cf5c9c8e834a83c50c002c635736666faad79f2bb952a7ea9b0194a1170ea WatchSource:0}: Error finding container 044cf5c9c8e834a83c50c002c635736666faad79f2bb952a7ea9b0194a1170ea: Status 404 returned error can't find the container with id 044cf5c9c8e834a83c50c002c635736666faad79f2bb952a7ea9b0194a1170ea
Sep 17 19:49:13.871383 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:13.871344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" event={"ID":"0bbde963-daf1-456f-b2e0-c712ab431aec","Type":"ContainerStarted","Data":"c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363"}
Sep 17 19:49:13.871810 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:13.871407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" event={"ID":"0bbde963-daf1-456f-b2e0-c712ab431aec","Type":"ContainerStarted","Data":"044cf5c9c8e834a83c50c002c635736666faad79f2bb952a7ea9b0194a1170ea"}
Sep 17 19:49:15.884761 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:15.884728    2583 generic.go:356] "Generic (PLEG): container finished" podID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerID="c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363" exitCode=0
Sep 17 19:49:15.885210 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:15.884796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" event={"ID":"0bbde963-daf1-456f-b2e0-c712ab431aec","Type":"ContainerDied","Data":"c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363"}
Sep 17 19:49:16.900057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:16.900021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" event={"ID":"0bbde963-daf1-456f-b2e0-c712ab431aec","Type":"ContainerStarted","Data":"bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9"}
Sep 17 19:49:16.924519 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:16.924435    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podStartSLOduration=2.716038581 podStartE2EDuration="4.924419529s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.208380948s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:49:16.92307446 +0000 UTC m=+2558.970954317" watchObservedRunningTime="2026-09-17 19:49:16.924419529 +0000 UTC m=+2558.972299386"
Sep 17 19:49:22.858306 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:22.858273    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:22.858713 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:22.858333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:49:22.859606 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:22.859573    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:49:32.858787 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:32.858746    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:49:36.158634 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:36.158599    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"]
Sep 17 19:49:42.858471 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:42.858429    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:49:51.182305 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.182202    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" containerID="cri-o://afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046" gracePeriod=45
Sep 17 19:49:51.295204 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.295156    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Sep 17 19:49:51.859183 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.858899    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:49:51.996919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.996842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-tmp-dir\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.996919 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.996876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-model-cache\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.997103 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.996933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-home\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.997103 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.996979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-dshm\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.997103 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997040    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-kserve-provision-location\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.997103 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997078    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ccc2a-118e-4382-9c6a-62830331e5a1-cabundle-cert\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.997324 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423ccc2a-118e-4382-9c6a-62830331e5a1-tls-certs\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.997324 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997138    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzwll\" (UniqueName: \"kubernetes.io/projected/423ccc2a-118e-4382-9c6a-62830331e5a1-kube-api-access-wzwll\") pod \"423ccc2a-118e-4382-9c6a-62830331e5a1\" (UID: \"423ccc2a-118e-4382-9c6a-62830331e5a1\") "
Sep 17 19:49:51.997324 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997216    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:51.997492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997314    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-home" (OuterVolumeSpecName: "home") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:51.997492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997451    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:51.997492 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997470    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:51.997645 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.997546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423ccc2a-118e-4382-9c6a-62830331e5a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:49:51.999839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:51.999793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-dshm" (OuterVolumeSpecName: "dshm") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:52.000132 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.000072    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423ccc2a-118e-4382-9c6a-62830331e5a1-kube-api-access-wzwll" (OuterVolumeSpecName: "kube-api-access-wzwll") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "kube-api-access-wzwll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:49:52.000335 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.000308    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423ccc2a-118e-4382-9c6a-62830331e5a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:49:52.009801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.009668    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:52.024066 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.024044    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "423ccc2a-118e-4382-9c6a-62830331e5a1" (UID: "423ccc2a-118e-4382-9c6a-62830331e5a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:52.098152 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.098128    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:52.098152 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.098152    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:52.098152 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.098163    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ccc2a-118e-4382-9c6a-62830331e5a1-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:52.098391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.098174    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423ccc2a-118e-4382-9c6a-62830331e5a1-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:52.098391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.098182    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwll\" (UniqueName: \"kubernetes.io/projected/423ccc2a-118e-4382-9c6a-62830331e5a1-kube-api-access-wzwll\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:52.098391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.098190    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423ccc2a-118e-4382-9c6a-62830331e5a1-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:49:52.104719 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.104679    2583 generic.go:356] "Generic (PLEG): container finished" podID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerID="afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046" exitCode=0
Sep 17 19:49:52.104827 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.104769    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"
Sep 17 19:49:52.104827 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.104788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" event={"ID":"423ccc2a-118e-4382-9c6a-62830331e5a1","Type":"ContainerDied","Data":"afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046"}
Sep 17 19:49:52.104945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.104874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2" event={"ID":"423ccc2a-118e-4382-9c6a-62830331e5a1","Type":"ContainerDied","Data":"b1f770c01f81a2dcd5440362e07c2a9524220e57fc4cb6eb28288b0f3e18b442"}
Sep 17 19:49:52.104945 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.104912    2583 scope.go:117] "RemoveContainer" containerID="afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046"
Sep 17 19:49:52.114489 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.114469    2583 scope.go:117] "RemoveContainer" containerID="6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237"
Sep 17 19:49:52.123333 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.123318    2583 scope.go:117] "RemoveContainer" containerID="afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046"
Sep 17 19:49:52.123636 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:49:52.123609    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046\": container with ID starting with afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046 not found: ID does not exist" containerID="afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046"
Sep 17 19:49:52.123702 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.123650    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046"} err="failed to get container status \"afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046\": rpc error: code = NotFound desc = could not find container \"afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046\": container with ID starting with afb2dce2c4e16ac1033102abbc208d6add04c343b9832721cca3936774a1c046 not found: ID does not exist"
Sep 17 19:49:52.123702 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.123673    2583 scope.go:117] "RemoveContainer" containerID="6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237"
Sep 17 19:49:52.123973 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:49:52.123955    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237\": container with ID starting with 6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237 not found: ID does not exist" containerID="6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237"
Sep 17 19:49:52.124026 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.123978    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237"} err="failed to get container status \"6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237\": rpc error: code = NotFound desc = could not find container \"6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237\": container with ID starting with 6a3509df333cbb9908890c4c749f9a338c991a814b33f02cdcf43e3a0c0e8237 not found: ID does not exist"
Sep 17 19:49:52.128484 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.128462    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"]
Sep 17 19:49:52.134008 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.133987    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-fccb4d955-sw9k2"]
Sep 17 19:49:52.256764 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.256677    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" path="/var/lib/kubelet/pods/423ccc2a-118e-4382-9c6a-62830331e5a1/volumes"
Sep 17 19:49:52.858834 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:49:52.858795    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:50:02.859113 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:02.859073    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:50:04.976482 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.976446    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"]
Sep 17 19:50:04.976937 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.976888    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main"
Sep 17 19:50:04.976937 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.976920    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main"
Sep 17 19:50:04.977016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.976940    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="storage-initializer"
Sep 17 19:50:04.977016 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.976946    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="storage-initializer"
Sep 17 19:50:04.977080 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.977024    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="423ccc2a-118e-4382-9c6a-62830331e5a1" containerName="main"
Sep 17 19:50:04.980296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.980274    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:04.982943 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.982921    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 19:50:04.983088 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.982925    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 19:50:04.983435 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.983412    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 19:50:04.983557 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.983455    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-q6wf6\""
Sep 17 19:50:04.983557 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.983511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 19:50:04.983742 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.983726    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:50:04.998453 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:04.998423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"]
Sep 17 19:50:05.111859 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.111823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-model-cache\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.112015 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.111872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14960032-c3aa-4102-958b-c073eef78691-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.112015 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.111914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.112015 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.111976    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-dshm\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.112015 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.111995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.112217 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.112025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-home\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.112217 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.112123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvxb\" (UniqueName: \"kubernetes.io/projected/14960032-c3aa-4102-958b-c073eef78691-kube-api-access-sqvxb\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.112217 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.112145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14960032-c3aa-4102-958b-c073eef78691-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213182 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvxb\" (UniqueName: \"kubernetes.io/projected/14960032-c3aa-4102-958b-c073eef78691-kube-api-access-sqvxb\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14960032-c3aa-4102-958b-c073eef78691-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-model-cache\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14960032-c3aa-4102-958b-c073eef78691-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213369 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213595 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-dshm\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213595 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213595 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-home\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.213921 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.213890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-home\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.214138 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.214111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.214431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.214406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14960032-c3aa-4102-958b-c073eef78691-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.214506 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.214427    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.214681 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.214659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-model-cache\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.216725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.216656    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-dshm\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.216968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.216945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14960032-c3aa-4102-958b-c073eef78691-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.221223 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.221202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvxb\" (UniqueName: \"kubernetes.io/projected/14960032-c3aa-4102-958b-c073eef78691-kube-api-access-sqvxb\") pod \"lora-multiple-adapters-test-kserve-7cb9c86c96-85v59\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.292234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.292174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:05.579070 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:05.579040    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"]
Sep 17 19:50:05.580052 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:50:05.580020    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14960032_c3aa_4102_958b_c073eef78691.slice/crio-fec8c146b97d1514980847371e77c6fcc49968ab0c640364f05c31ea362f29da WatchSource:0}: Error finding container fec8c146b97d1514980847371e77c6fcc49968ab0c640364f05c31ea362f29da: Status 404 returned error can't find the container with id fec8c146b97d1514980847371e77c6fcc49968ab0c640364f05c31ea362f29da
Sep 17 19:50:06.189034 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:06.188992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" event={"ID":"14960032-c3aa-4102-958b-c073eef78691","Type":"ContainerStarted","Data":"6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f"}
Sep 17 19:50:06.189543 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:06.189063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" event={"ID":"14960032-c3aa-4102-958b-c073eef78691","Type":"ContainerStarted","Data":"fec8c146b97d1514980847371e77c6fcc49968ab0c640364f05c31ea362f29da"}
Sep 17 19:50:09.212450 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:09.212415    2583 generic.go:356] "Generic (PLEG): container finished" podID="14960032-c3aa-4102-958b-c073eef78691" containerID="6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f" exitCode=0
Sep 17 19:50:09.212958 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:09.212494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" event={"ID":"14960032-c3aa-4102-958b-c073eef78691","Type":"ContainerDied","Data":"6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f"}
Sep 17 19:50:10.220296 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:10.220261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" event={"ID":"14960032-c3aa-4102-958b-c073eef78691","Type":"ContainerStarted","Data":"5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033"}
Sep 17 19:50:10.244938 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:10.244878    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podStartSLOduration=3.144210198 podStartE2EDuration="6.244856661s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.100646463s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:50:10.242169449 +0000 UTC m=+2612.290049308" watchObservedRunningTime="2026-09-17 19:50:10.244856661 +0000 UTC m=+2612.292736523"
Sep 17 19:50:10.293122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:10.293086    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:12.858397 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:12.858342    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:50:15.292558 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:15.292500    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:50:15.294195 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:15.294163    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:50:22.859062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:22.859007    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:50:25.293253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:25.293205    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:50:32.858826 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:32.858772    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:50:35.293574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:35.293528    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:50:42.859090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:42.859042    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:50:45.293227 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:45.293169    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:50:52.859424 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:52.859367    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:50:55.293456 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:50:55.293409    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:51:02.859299 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:02.859249    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 17 19:51:05.292671 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:05.292629    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:51:12.868708 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:12.868650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:51:12.876511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:12.876463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:51:15.293238 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:15.293187    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:51:25.302651 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:25.302571    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:51:25.310385 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:25.310350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:51:34.036348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:34.036311    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"]
Sep 17 19:51:49.059843 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:49.059798    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main" containerID="cri-o://bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9" gracePeriod=45
Sep 17 19:51:50.052206 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.052182    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:51:50.195835 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195808    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbsfm\" (UniqueName: \"kubernetes.io/projected/0bbde963-daf1-456f-b2e0-c712ab431aec-kube-api-access-tbsfm\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.195835 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-tmp-dir\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.196343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195867    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-kserve-provision-location\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.196343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195907    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-home\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.196343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195937    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-dshm\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.196343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbde963-daf1-456f-b2e0-c712ab431aec-tls-certs\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.196343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195973    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-model-cache\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.196343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.195996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbde963-daf1-456f-b2e0-c712ab431aec-cabundle-cert\") pod \"0bbde963-daf1-456f-b2e0-c712ab431aec\" (UID: \"0bbde963-daf1-456f-b2e0-c712ab431aec\") "
Sep 17 19:51:50.196857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.196552    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.196857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.196606    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbde963-daf1-456f-b2e0-c712ab431aec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:51:50.196857 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.196634    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-home" (OuterVolumeSpecName: "home") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.198460 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.198434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-dshm" (OuterVolumeSpecName: "dshm") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.198628 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.198605    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbde963-daf1-456f-b2e0-c712ab431aec-kube-api-access-tbsfm" (OuterVolumeSpecName: "kube-api-access-tbsfm") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "kube-api-access-tbsfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:51:50.198832 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.198805    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbde963-daf1-456f-b2e0-c712ab431aec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:51:50.215437 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.215397    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.227276 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.227244    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bbde963-daf1-456f-b2e0-c712ab431aec" (UID: "0bbde963-daf1-456f-b2e0-c712ab431aec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:51:50.297426 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297392    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbde963-daf1-456f-b2e0-c712ab431aec-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.297426 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297421    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.297599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297435    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bbde963-daf1-456f-b2e0-c712ab431aec-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.297599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297449    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbsfm\" (UniqueName: \"kubernetes.io/projected/0bbde963-daf1-456f-b2e0-c712ab431aec-kube-api-access-tbsfm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.297599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297486    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.297599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297502    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.297599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297515    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.297599 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.297526    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbde963-daf1-456f-b2e0-c712ab431aec-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:51:50.863144 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.863093    2583 generic.go:356] "Generic (PLEG): container finished" podID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerID="bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9" exitCode=0
Sep 17 19:51:50.863323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.863156    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"
Sep 17 19:51:50.863323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.863178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" event={"ID":"0bbde963-daf1-456f-b2e0-c712ab431aec","Type":"ContainerDied","Data":"bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9"}
Sep 17 19:51:50.863323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.863234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82" event={"ID":"0bbde963-daf1-456f-b2e0-c712ab431aec","Type":"ContainerDied","Data":"044cf5c9c8e834a83c50c002c635736666faad79f2bb952a7ea9b0194a1170ea"}
Sep 17 19:51:50.863323 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.863264    2583 scope.go:117] "RemoveContainer" containerID="bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9"
Sep 17 19:51:50.876871 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.876840    2583 scope.go:117] "RemoveContainer" containerID="c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363"
Sep 17 19:51:50.889031 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.889004    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"]
Sep 17 19:51:50.892159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.892135    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6b4mm82"]
Sep 17 19:51:50.892994 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.892966    2583 scope.go:117] "RemoveContainer" containerID="bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9"
Sep 17 19:51:50.893286 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:51:50.893265    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9\": container with ID starting with bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9 not found: ID does not exist" containerID="bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9"
Sep 17 19:51:50.893344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.893292    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9"} err="failed to get container status \"bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9\": rpc error: code = NotFound desc = could not find container \"bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9\": container with ID starting with bf1e2f2e92a19811145b555c4fb8c2ce09e3899e1d6289602808771ca63df2b9 not found: ID does not exist"
Sep 17 19:51:50.893344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.893310    2583 scope.go:117] "RemoveContainer" containerID="c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363"
Sep 17 19:51:50.893569 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:51:50.893554    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363\": container with ID starting with c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363 not found: ID does not exist" containerID="c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363"
Sep 17 19:51:50.893613 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:50.893574    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363"} err="failed to get container status \"c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363\": rpc error: code = NotFound desc = could not find container \"c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363\": container with ID starting with c8e11a2a64ab3fa0e0ec1a551b45cda790fd2cb41a1e5d9992197530e5d9b363 not found: ID does not exist"
Sep 17 19:51:52.262444 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:52.262412    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" path="/var/lib/kubelet/pods/0bbde963-daf1-456f-b2e0-c712ab431aec/volumes"
Sep 17 19:51:52.752130 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:52.752080    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7cb9c86c96-85v59_14960032-c3aa-4102-958b-c073eef78691/storage-initializer/0.log"
Sep 17 19:51:52.764645 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:52.764609    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7cb9c86c96-85v59_14960032-c3aa-4102-958b-c073eef78691/main/0.log"
Sep 17 19:51:53.897839 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:51:53.897806    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"]
Sep 17 19:52:08.921856 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:08.921808    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" containerID="cri-o://5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033" gracePeriod=45
Sep 17 19:52:09.303283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.303247    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 17 19:52:09.580674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.580651    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:52:09.761197 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761168    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-tmp-dir\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761210    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14960032-c3aa-4102-958b-c073eef78691-tls-certs\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761242    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14960032-c3aa-4102-958b-c073eef78691-cabundle-cert\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761263    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-kserve-provision-location\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761315    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-model-cache\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761331 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761331    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-dshm\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761587 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761354    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqvxb\" (UniqueName: \"kubernetes.io/projected/14960032-c3aa-4102-958b-c073eef78691-kube-api-access-sqvxb\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761587 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761370    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-home\") pod \"14960032-c3aa-4102-958b-c073eef78691\" (UID: \"14960032-c3aa-4102-958b-c073eef78691\") "
Sep 17 19:52:09.761807 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761776    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-model-cache" (OuterVolumeSpecName: "model-cache") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:52:09.761880 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14960032-c3aa-4102-958b-c073eef78691-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:52:09.761934 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.761907    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-home" (OuterVolumeSpecName: "home") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:52:09.764250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.764220    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14960032-c3aa-4102-958b-c073eef78691-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:52:09.764250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.764226    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-dshm" (OuterVolumeSpecName: "dshm") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:52:09.764250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.764242    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14960032-c3aa-4102-958b-c073eef78691-kube-api-access-sqvxb" (OuterVolumeSpecName: "kube-api-access-sqvxb") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "kube-api-access-sqvxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:52:09.777427 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.777381    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:52:09.794256 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.794225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14960032-c3aa-4102-958b-c073eef78691" (UID: "14960032-c3aa-4102-958b-c073eef78691"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:52:09.862833 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862781    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.862833 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862803    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.862833 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862813    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvxb\" (UniqueName: \"kubernetes.io/projected/14960032-c3aa-4102-958b-c073eef78691-kube-api-access-sqvxb\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.862833 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862824    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.863013 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862853    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.863013 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862863    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14960032-c3aa-4102-958b-c073eef78691-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.863013 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862871    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14960032-c3aa-4102-958b-c073eef78691-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.863013 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.862879    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14960032-c3aa-4102-958b-c073eef78691-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:09.981936 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.981909    2583 generic.go:356] "Generic (PLEG): container finished" podID="14960032-c3aa-4102-958b-c073eef78691" containerID="5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033" exitCode=0
Sep 17 19:52:09.982266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.981969    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"
Sep 17 19:52:09.982266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.982000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" event={"ID":"14960032-c3aa-4102-958b-c073eef78691","Type":"ContainerDied","Data":"5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033"}
Sep 17 19:52:09.982266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.982063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59" event={"ID":"14960032-c3aa-4102-958b-c073eef78691","Type":"ContainerDied","Data":"fec8c146b97d1514980847371e77c6fcc49968ab0c640364f05c31ea362f29da"}
Sep 17 19:52:09.982266 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.982086    2583 scope.go:117] "RemoveContainer" containerID="5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033"
Sep 17 19:52:09.992173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:09.992154    2583 scope.go:117] "RemoveContainer" containerID="6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f"
Sep 17 19:52:10.000993 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:10.000977    2583 scope.go:117] "RemoveContainer" containerID="5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033"
Sep 17 19:52:10.001292 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:52:10.001271    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033\": container with ID starting with 5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033 not found: ID does not exist" containerID="5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033"
Sep 17 19:52:10.001348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:10.001299    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033"} err="failed to get container status \"5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033\": rpc error: code = NotFound desc = could not find container \"5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033\": container with ID starting with 5d6be8cbb4e8efd47aa7addae2baad8dd282fd9c365c46368c7f611e128d8033 not found: ID does not exist"
Sep 17 19:52:10.001348 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:10.001317    2583 scope.go:117] "RemoveContainer" containerID="6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f"
Sep 17 19:52:10.001584 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:52:10.001564    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f\": container with ID starting with 6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f not found: ID does not exist" containerID="6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f"
Sep 17 19:52:10.001622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:10.001591    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f"} err="failed to get container status \"6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f\": rpc error: code = NotFound desc = could not find container \"6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f\": container with ID starting with 6f30f838511fda33bc0d63fea1c1247280a7e88370bda5972894b47c4e993e2f not found: ID does not exist"
Sep 17 19:52:10.007458 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:10.007433    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"]
Sep 17 19:52:10.011495 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:10.011471    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7cb9c86c96-85v59"]
Sep 17 19:52:10.255793 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:10.255723    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14960032-c3aa-4102-958b-c073eef78691" path="/var/lib/kubelet/pods/14960032-c3aa-4102-958b-c073eef78691/volumes"
Sep 17 19:52:17.333412 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333367    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"]
Sep 17 19:52:17.333867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333805    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="storage-initializer"
Sep 17 19:52:17.333867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333820    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="storage-initializer"
Sep 17 19:52:17.333867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333833    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main"
Sep 17 19:52:17.333867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333839    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main"
Sep 17 19:52:17.333867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333859    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="storage-initializer"
Sep 17 19:52:17.333867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333865    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="storage-initializer"
Sep 17 19:52:17.333867 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333871    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main"
Sep 17 19:52:17.334097 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333876    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main"
Sep 17 19:52:17.334097 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333936    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="14960032-c3aa-4102-958b-c073eef78691" containerName="main"
Sep 17 19:52:17.334097 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.333947    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bbde963-daf1-456f-b2e0-c712ab431aec" containerName="main"
Sep 17 19:52:17.339057 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.339036    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:17.341924 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.341896    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 19:52:17.341924 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.341913    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-qzp4d\""
Sep 17 19:52:17.343012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.342981    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 19:52:17.343159 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.343125    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 19:52:17.345982 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.345953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"]
Sep 17 19:52:17.527466 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.527438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjzq\" (UniqueName: \"kubernetes.io/projected/7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e-kube-api-access-hxjzq\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:17.628047 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.627973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjzq\" (UniqueName: \"kubernetes.io/projected/7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e-kube-api-access-hxjzq\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:17.637137 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:17.637107    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjzq\" (UniqueName: \"kubernetes.io/projected/7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e-kube-api-access-hxjzq\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.006567 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.006534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.012543 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.012519    2583 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.107802 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.107773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.107931 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.107906    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.110504 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.110486    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") DevicePath \"csi-742d4d5481f5ab6fa99cefcb551112ce2deedeabb0346cff49e0aa2c2041293a\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.327662 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.327634    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/91759b7e8927912fec78e1b02c8102348c57062d8d397d594d13f7d95311677f/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.341670 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.341647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"e2e-pvc-lora-download-54hlr\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.549825 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.549800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:27.823477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:27.823447    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"]
Sep 17 19:52:27.825065 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:52:27.825035    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e7e7e1f_0c46_4fc1_a2df_3b95752d1b5e.slice/crio-8ca635ab19d282e8374076f73e489308c60aeb48013e53986412c2ba447cce46 WatchSource:0}: Error finding container 8ca635ab19d282e8374076f73e489308c60aeb48013e53986412c2ba447cce46: Status 404 returned error can't find the container with id 8ca635ab19d282e8374076f73e489308c60aeb48013e53986412c2ba447cce46
Sep 17 19:52:28.085941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:28.085846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr" event={"ID":"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e","Type":"ContainerStarted","Data":"57ae05628c9e367c69279f9f49002de1f6a5c85568aea9370f65fb34628facdb"}
Sep 17 19:52:28.085941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:28.085887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr" event={"ID":"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e","Type":"ContainerStarted","Data":"8ca635ab19d282e8374076f73e489308c60aeb48013e53986412c2ba447cce46"}
Sep 17 19:52:28.102005 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:28.101961    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr" podStartSLOduration=14.101947441 podStartE2EDuration="14.101947441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:52:28.100831311 +0000 UTC m=+2750.148711169" watchObservedRunningTime="2026-09-17 19:52:28.101947441 +0000 UTC m=+2750.149827329"
Sep 17 19:52:30.101642 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:30.101610    2583 generic.go:356] "Generic (PLEG): container finished" podID="7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e" containerID="57ae05628c9e367c69279f9f49002de1f6a5c85568aea9370f65fb34628facdb" exitCode=0
Sep 17 19:52:30.102000 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:30.101674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr" event={"ID":"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e","Type":"ContainerDied","Data":"57ae05628c9e367c69279f9f49002de1f6a5c85568aea9370f65fb34628facdb"}
Sep 17 19:52:31.246783 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.246755    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:31.442940 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.442854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxjzq\" (UniqueName: \"kubernetes.io/projected/7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e-kube-api-access-hxjzq\") pod \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") "
Sep 17 19:52:31.443105 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.443036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\" (UID: \"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e\") "
Sep 17 19:52:31.445446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.445419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e-kube-api-access-hxjzq" (OuterVolumeSpecName: "kube-api-access-hxjzq") pod "7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e" (UID: "7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e"). InnerVolumeSpecName "kube-api-access-hxjzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:52:31.447239 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.447215    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27" (OuterVolumeSpecName: "model-storage") pod "7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e" (UID: "7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e"). InnerVolumeSpecName "pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:52:31.544136 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.544104    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxjzq\" (UniqueName: \"kubernetes.io/projected/7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e-kube-api-access-hxjzq\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:52:31.544293 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.544163    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") on node \"ip-10-0-128-131.ec2.internal\" "
Sep 17 19:52:31.545523 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.545498    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"]
Sep 17 19:52:31.552032 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.552000    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e" containerName="storage-initializer"
Sep 17 19:52:31.552199 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.552184    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e" containerName="storage-initializer"
Sep 17 19:52:31.552699 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.552648    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e" containerName="storage-initializer"
Sep 17 19:52:31.557536 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.557505    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.560307 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.560287    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cn4l7\""
Sep 17 19:52:31.560795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.560769    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:52:31.560894 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.560776    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 19:52:31.560894 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.560887    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 19:52:31.561019 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.560817    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 19:52:31.561019 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.560785    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 19:52:31.561556 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.561538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"]
Sep 17 19:52:31.645619 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.645759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.645759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.645759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b577b0-12d5-439d-ba00-3515d673997c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.645759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6km\" (UniqueName: \"kubernetes.io/projected/56b577b0-12d5-439d-ba00-3515d673997c-kube-api-access-xk6km\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.645913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.645913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56b577b0-12d5-439d-ba00-3515d673997c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.645913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.645897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.746912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.746849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.746912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.746882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.746912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.746906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b577b0-12d5-439d-ba00-3515d673997c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6km\" (UniqueName: \"kubernetes.io/projected/56b577b0-12d5-439d-ba00-3515d673997c-kube-api-access-xk6km\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747249 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56b577b0-12d5-439d-ba00-3515d673997c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747249 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747249 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747452 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747568 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.747879 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.747862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56b577b0-12d5-439d-ba00-3515d673997c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.749379 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.749364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.749633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.749615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b577b0-12d5-439d-ba00-3515d673997c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.755363 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.755335    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6km\" (UniqueName: \"kubernetes.io/projected/56b577b0-12d5-439d-ba00-3515d673997c-kube-api-access-xk6km\") pod \"custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:31.876277 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:31.876246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:32.114555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:32.114522    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"
Sep 17 19:52:32.114751 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:32.114521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr" event={"ID":"7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e","Type":"ContainerDied","Data":"8ca635ab19d282e8374076f73e489308c60aeb48013e53986412c2ba447cce46"}
Sep 17 19:52:32.114751 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:32.114644    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ca635ab19d282e8374076f73e489308c60aeb48013e53986412c2ba447cce46"
Sep 17 19:52:32.157773 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:32.157751    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"]
Sep 17 19:52:32.159546 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:52:32.159528    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b577b0_12d5_439d_ba00_3515d673997c.slice/crio-113155c2170b2c382f51136bbbd6096c14eef3b58918c863e32f03032ee89d8d WatchSource:0}: Error finding container 113155c2170b2c382f51136bbbd6096c14eef3b58918c863e32f03032ee89d8d: Status 404 returned error can't find the container with id 113155c2170b2c382f51136bbbd6096c14eef3b58918c863e32f03032ee89d8d
Sep 17 19:52:32.523507 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:32.523481    2583 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27") on node "ip-10-0-128-131.ec2.internal"
Sep 17 19:52:32.555508 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:32.555476    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"csi-742d4d5481f5ab6fa99cefcb551112ce2deedeabb0346cff49e0aa2c2041293a\""
Sep 17 19:52:33.124743 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:33.124681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" event={"ID":"56b577b0-12d5-439d-ba00-3515d673997c","Type":"ContainerStarted","Data":"58e9a6a040b95a52296966e26277e8bfc7903dbbaaebf38b5b6095be4c3d1278"}
Sep 17 19:52:33.124924 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:33.124768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" event={"ID":"56b577b0-12d5-439d-ba00-3515d673997c","Type":"ContainerStarted","Data":"113155c2170b2c382f51136bbbd6096c14eef3b58918c863e32f03032ee89d8d"}
Sep 17 19:52:35.143163 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:35.143128    2583 generic.go:356] "Generic (PLEG): container finished" podID="56b577b0-12d5-439d-ba00-3515d673997c" containerID="58e9a6a040b95a52296966e26277e8bfc7903dbbaaebf38b5b6095be4c3d1278" exitCode=0
Sep 17 19:52:35.143794 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:35.143204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" event={"ID":"56b577b0-12d5-439d-ba00-3515d673997c","Type":"ContainerDied","Data":"58e9a6a040b95a52296966e26277e8bfc7903dbbaaebf38b5b6095be4c3d1278"}
Sep 17 19:52:36.153110 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:36.153075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" event={"ID":"56b577b0-12d5-439d-ba00-3515d673997c","Type":"ContainerStarted","Data":"b9bc42ce79200b05061e72280036f75dacc2936f1a124d20cafa8ad5e6419f61"}
Sep 17 19:52:36.178164 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:36.178105    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podStartSLOduration=2.965912382 podStartE2EDuration="5.178090231s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.212177849s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:52:36.175992354 +0000 UTC m=+2758.223872224" watchObservedRunningTime="2026-09-17 19:52:36.178090231 +0000 UTC m=+2758.225970087"
Sep 17 19:52:41.877203 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:41.877151    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:41.877736 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:41.877256    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:52:41.878869 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:41.878840    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:52:44.869347 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.869309    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"]
Sep 17 19:52:44.872119 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.872096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.874784 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.874761    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 19:52:44.874913 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.874880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-qzp4d\""
Sep 17 19:52:44.875069 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.875047    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 19:52:44.875314 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.875290    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:52:44.875450 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.875357    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 19:52:44.875915 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.875893    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 19:52:44.885541 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.885521    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"]
Sep 17 19:52:44.961054 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.961208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-dshm\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.961208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.961208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wsm2\" (UniqueName: \"kubernetes.io/projected/42747856-4650-470d-b335-c372279ec005-kube-api-access-7wsm2\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.961208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42747856-4650-470d-b335-c372279ec005-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.961208 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42747856-4650-470d-b335-c372279ec005-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.961378 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-home\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:44.961378 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:44.961278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062107 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-dshm\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062107 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wsm2\" (UniqueName: \"kubernetes.io/projected/42747856-4650-470d-b335-c372279ec005-kube-api-access-7wsm2\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42747856-4650-470d-b335-c372279ec005-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42747856-4650-470d-b335-c372279ec005-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-home\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062613 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062787 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-home\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.062787 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.062775    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.063250 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.063227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.063304 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.063245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42747856-4650-470d-b335-c372279ec005-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.063450 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.063432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.065005 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.064987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-dshm\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.065113 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.065037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42747856-4650-470d-b335-c372279ec005-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:45.070814 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:45.070790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wsm2\" (UniqueName: \"kubernetes.io/projected/42747856-4650-470d-b335-c372279ec005-kube-api-access-7wsm2\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.586844 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.586805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.598056 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.598018    2583 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.687321 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.687291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.687458 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.687403    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.690635 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.690618    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") DevicePath \"csi-742d4d5481f5ab6fa99cefcb551112ce2deedeabb0346cff49e0aa2c2041293a\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.779872 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.779828    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/91759b7e8927912fec78e1b02c8102348c57062d8d397d594d13f7d95311677f/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.799046 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.799019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr\" (UID: \"42747856-4650-470d-b335-c372279ec005\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:47.883743 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:47.883660    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:48.164904 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:48.164875    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"]
Sep 17 19:52:48.166336 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:52:48.166297    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42747856_4650_470d_b335_c372279ec005.slice/crio-11f8bef3739739a95badd3137de4697772aabe84a5ffe91402222aa8eb4ea453 WatchSource:0}: Error finding container 11f8bef3739739a95badd3137de4697772aabe84a5ffe91402222aa8eb4ea453: Status 404 returned error can't find the container with id 11f8bef3739739a95badd3137de4697772aabe84a5ffe91402222aa8eb4ea453
Sep 17 19:52:48.229228 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:48.229193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" event={"ID":"42747856-4650-470d-b335-c372279ec005","Type":"ContainerStarted","Data":"c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00"}
Sep 17 19:52:48.229341 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:48.229261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" event={"ID":"42747856-4650-470d-b335-c372279ec005","Type":"ContainerStarted","Data":"11f8bef3739739a95badd3137de4697772aabe84a5ffe91402222aa8eb4ea453"}
Sep 17 19:52:51.250033 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:51.250003    2583 generic.go:356] "Generic (PLEG): container finished" podID="42747856-4650-470d-b335-c372279ec005" containerID="c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00" exitCode=0
Sep 17 19:52:51.250559 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:51.250060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" event={"ID":"42747856-4650-470d-b335-c372279ec005","Type":"ContainerDied","Data":"c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00"}
Sep 17 19:52:51.877449 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:51.877405    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:52:52.259731 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:52.259623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" event={"ID":"42747856-4650-470d-b335-c372279ec005","Type":"ContainerStarted","Data":"6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899"}
Sep 17 19:52:52.285431 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:52.285378    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podStartSLOduration=6.00911563 podStartE2EDuration="8.285362207s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.276246577s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:52:52.28213462 +0000 UTC m=+2774.330014476" watchObservedRunningTime="2026-09-17 19:52:52.285362207 +0000 UTC m=+2774.333242098"
Sep 17 19:52:52.884538 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:52.884494    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:57.884371 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:57.884333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:52:57.885725 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:52:57.885674    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 17 19:53:01.877189 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:01.877147    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:53:07.884581 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:07.884533    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 17 19:53:11.876891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:11.876831    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:53:17.884794 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:17.884744    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 17 19:53:21.877375 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:21.877326    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:53:27.884622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:27.884576    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 17 19:53:31.877003 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:31.876956    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:53:37.884708 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:37.884654    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 17 19:53:41.877215 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:41.877174    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:53:47.884115 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:47.884064    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 17 19:53:51.877542 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:51.877490    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:53:57.884830 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:53:57.884783    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Sep 17 19:54:01.877098 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:01.877051    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:54:07.895090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:07.895052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:54:07.904452 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:07.904428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:54:11.876795 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:11.876743    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:54:21.877024 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:21.876937    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:54:31.877827 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:31.877765    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 17 19:54:41.887132 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:41.887097    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:54:41.894951 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:41.894926    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:54:49.768329 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:49.768297    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr_42747856-4650-470d-b335-c372279ec005/storage-initializer/0.log"
Sep 17 19:54:49.778935 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:49.778897    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr_42747856-4650-470d-b335-c372279ec005/main/0.log"
Sep 17 19:54:50.878495 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:50.878453    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"]
Sep 17 19:54:54.407139 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:54:54.407104    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"]
Sep 17 19:55:05.902438 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:05.902371    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main" containerID="cri-o://6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899" gracePeriod=45
Sep 17 19:55:06.562372 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.562349    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:55:06.679909 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.679844    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-home\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.679909 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.679880    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-dshm\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.679909 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.679903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42747856-4650-470d-b335-c372279ec005-tls-certs\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.680171 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.680171 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680079    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-tmp-dir\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.680171 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680118    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wsm2\" (UniqueName: \"kubernetes.io/projected/42747856-4650-470d-b335-c372279ec005-kube-api-access-7wsm2\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.680346 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680171    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-kserve-provision-location\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.680346 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680237    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-model-cache\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.680346 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-home" (OuterVolumeSpecName: "home") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:06.680346 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680266    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42747856-4650-470d-b335-c372279ec005-cabundle-cert\") pod \"42747856-4650-470d-b335-c372279ec005\" (UID: \"42747856-4650-470d-b335-c372279ec005\") "
Sep 17 19:55:06.680590 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680545    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42747856-4650-470d-b335-c372279ec005-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:06.680825 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680806    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-model-cache" (OuterVolumeSpecName: "model-cache") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:06.680893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680848    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42747856-4650-470d-b335-c372279ec005-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.680893 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.680863    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.682442 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.682412    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-dshm" (OuterVolumeSpecName: "dshm") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:06.682676 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.682643    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42747856-4650-470d-b335-c372279ec005-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:06.683234 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.683206    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42747856-4650-470d-b335-c372279ec005-kube-api-access-7wsm2" (OuterVolumeSpecName: "kube-api-access-7wsm2") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "kube-api-access-7wsm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:06.685418 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.685391    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:55:06.692257 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.692235    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:06.702966 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.702941    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42747856-4650-470d-b335-c372279ec005" (UID: "42747856-4650-470d-b335-c372279ec005"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:06.781515 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.781484    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.781515 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.781512    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42747856-4650-470d-b335-c372279ec005-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.781667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.781549    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") on node \"ip-10-0-128-131.ec2.internal\" "
Sep 17 19:55:06.781667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.781563    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.781667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.781577    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wsm2\" (UniqueName: \"kubernetes.io/projected/42747856-4650-470d-b335-c372279ec005-kube-api-access-7wsm2\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.781667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.781591    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.781667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.781604    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42747856-4650-470d-b335-c372279ec005-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:06.791982 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.791968    2583 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27") on node "ip-10-0-128-131.ec2.internal"
Sep 17 19:55:06.882638 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:06.882612    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-f2aa20ff-66cf-485a-8175-0b0e4463381b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ab3374a177bc8b27\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"csi-742d4d5481f5ab6fa99cefcb551112ce2deedeabb0346cff49e0aa2c2041293a\""
Sep 17 19:55:07.113803 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.113771    2583 generic.go:356] "Generic (PLEG): container finished" podID="42747856-4650-470d-b335-c372279ec005" containerID="6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899" exitCode=0
Sep 17 19:55:07.114170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.113822    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"
Sep 17 19:55:07.114170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.113849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" event={"ID":"42747856-4650-470d-b335-c372279ec005","Type":"ContainerDied","Data":"6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899"}
Sep 17 19:55:07.114170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.113905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr" event={"ID":"42747856-4650-470d-b335-c372279ec005","Type":"ContainerDied","Data":"11f8bef3739739a95badd3137de4697772aabe84a5ffe91402222aa8eb4ea453"}
Sep 17 19:55:07.114170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.113927    2583 scope.go:117] "RemoveContainer" containerID="6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899"
Sep 17 19:55:07.124030 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.124013    2583 scope.go:117] "RemoveContainer" containerID="c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00"
Sep 17 19:55:07.132452 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.132436    2583 scope.go:117] "RemoveContainer" containerID="6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899"
Sep 17 19:55:07.132675 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:55:07.132656    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899\": container with ID starting with 6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899 not found: ID does not exist" containerID="6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899"
Sep 17 19:55:07.132781 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.132704    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899"} err="failed to get container status \"6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899\": rpc error: code = NotFound desc = could not find container \"6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899\": container with ID starting with 6d303389a0854571ddace24a797e8241c62a5866d81b0593d03dffb871e6b899 not found: ID does not exist"
Sep 17 19:55:07.132781 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.132727    2583 scope.go:117] "RemoveContainer" containerID="c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00"
Sep 17 19:55:07.132979 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:55:07.132962    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00\": container with ID starting with c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00 not found: ID does not exist" containerID="c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00"
Sep 17 19:55:07.133027 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.132983    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00"} err="failed to get container status \"c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00\": rpc error: code = NotFound desc = could not find container \"c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00\": container with ID starting with c2da8eb513cf20c8d4e9d0fd94b0f3ac2b1ca50b487d4cf95826b2946e0f3c00 not found: ID does not exist"
Sep 17 19:55:07.137787 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.137762    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"]
Sep 17 19:55:07.141322 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:07.141303    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-594fd8b779-mn7hr"]
Sep 17 19:55:08.256198 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:08.256161    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42747856-4650-470d-b335-c372279ec005" path="/var/lib/kubelet/pods/42747856-4650-470d-b335-c372279ec005/volumes"
Sep 17 19:55:09.433450 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:09.433394    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main" containerID="cri-o://b9bc42ce79200b05061e72280036f75dacc2936f1a124d20cafa8ad5e6419f61" gracePeriod=45
Sep 17 19:55:10.136941 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.136906    2583 generic.go:356] "Generic (PLEG): container finished" podID="56b577b0-12d5-439d-ba00-3515d673997c" containerID="b9bc42ce79200b05061e72280036f75dacc2936f1a124d20cafa8ad5e6419f61" exitCode=0
Sep 17 19:55:10.137098 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.136975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" event={"ID":"56b577b0-12d5-439d-ba00-3515d673997c","Type":"ContainerDied","Data":"b9bc42ce79200b05061e72280036f75dacc2936f1a124d20cafa8ad5e6419f61"}
Sep 17 19:55:10.186619 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.186594    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:55:10.312825 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.312799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk6km\" (UniqueName: \"kubernetes.io/projected/56b577b0-12d5-439d-ba00-3515d673997c-kube-api-access-xk6km\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.312975 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.312841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56b577b0-12d5-439d-ba00-3515d673997c-cabundle-cert\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.312975 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.312861    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-model-cache\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.313107 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.312982    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-dshm\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.313107 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-tmp-dir\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.313107 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-kserve-provision-location\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.313107 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-home\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.313107 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313076    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-model-cache" (OuterVolumeSpecName: "model-cache") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.313372 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313123    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b577b0-12d5-439d-ba00-3515d673997c-tls-certs\") pod \"56b577b0-12d5-439d-ba00-3515d673997c\" (UID: \"56b577b0-12d5-439d-ba00-3515d673997c\") "
Sep 17 19:55:10.313372 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313344    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b577b0-12d5-439d-ba00-3515d673997c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:10.313477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313416    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56b577b0-12d5-439d-ba00-3515d673997c-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.313477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313435    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.313850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.313756    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-home" (OuterVolumeSpecName: "home") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.315427 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.315400    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b577b0-12d5-439d-ba00-3515d673997c-kube-api-access-xk6km" (OuterVolumeSpecName: "kube-api-access-xk6km") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "kube-api-access-xk6km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:10.315540 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.315503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-dshm" (OuterVolumeSpecName: "dshm") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.315800 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.315780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b577b0-12d5-439d-ba00-3515d673997c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:10.324401 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.324363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.338974 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.338941    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56b577b0-12d5-439d-ba00-3515d673997c" (UID: "56b577b0-12d5-439d-ba00-3515d673997c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:10.414425 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.414389    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.414425 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.414422    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.414637 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.414436    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.414637 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.414445    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b577b0-12d5-439d-ba00-3515d673997c-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.414637 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.414453    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b577b0-12d5-439d-ba00-3515d673997c-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:10.414637 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:10.414462    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6km\" (UniqueName: \"kubernetes.io/projected/56b577b0-12d5-439d-ba00-3515d673997c-kube-api-access-xk6km\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:11.144124 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:11.144052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s" event={"ID":"56b577b0-12d5-439d-ba00-3515d673997c","Type":"ContainerDied","Data":"113155c2170b2c382f51136bbbd6096c14eef3b58918c863e32f03032ee89d8d"}
Sep 17 19:55:11.144124 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:11.144130    2583 scope.go:117] "RemoveContainer" containerID="b9bc42ce79200b05061e72280036f75dacc2936f1a124d20cafa8ad5e6419f61"
Sep 17 19:55:11.144621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:11.144065    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"
Sep 17 19:55:11.154627 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:11.154607    2583 scope.go:117] "RemoveContainer" containerID="58e9a6a040b95a52296966e26277e8bfc7903dbbaaebf38b5b6095be4c3d1278"
Sep 17 19:55:11.173955 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:11.173926    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"]
Sep 17 19:55:11.180788 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:11.180766    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-db9f67448-kxr4s"]
Sep 17 19:55:12.254802 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:12.254770    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b577b0-12d5-439d-ba00-3515d673997c" path="/var/lib/kubelet/pods/56b577b0-12d5-439d-ba00-3515d673997c/volumes"
Sep 17 19:55:14.395699 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.395647    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"]
Sep 17 19:55:14.396072 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396038    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main"
Sep 17 19:55:14.396072 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396050    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main"
Sep 17 19:55:14.396072 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396062    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main"
Sep 17 19:55:14.396072 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396067    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main"
Sep 17 19:55:14.396209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396085    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42747856-4650-470d-b335-c372279ec005" containerName="storage-initializer"
Sep 17 19:55:14.396209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396090    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="42747856-4650-470d-b335-c372279ec005" containerName="storage-initializer"
Sep 17 19:55:14.396209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396104    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="storage-initializer"
Sep 17 19:55:14.396209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396110    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="storage-initializer"
Sep 17 19:55:14.396209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396162    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="42747856-4650-470d-b335-c372279ec005" containerName="main"
Sep 17 19:55:14.396209 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.396173    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b577b0-12d5-439d-ba00-3515d673997c" containerName="main"
Sep 17 19:55:14.401075 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.401056    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:14.403952 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.403930    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 19:55:14.405344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.405187    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-sbbt9\""
Sep 17 19:55:14.405344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.405208    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 19:55:14.405344 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.405199    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 19:55:14.408122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.408094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"]
Sep 17 19:55:14.546433 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.546400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5kf5\" (UniqueName: \"kubernetes.io/projected/c89feb69-b2cf-4886-9295-e1657179a88b-kube-api-access-g5kf5\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:14.647615 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.647523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5kf5\" (UniqueName: \"kubernetes.io/projected/c89feb69-b2cf-4886-9295-e1657179a88b-kube-api-access-g5kf5\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:14.657674 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:14.657643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5kf5\" (UniqueName: \"kubernetes.io/projected/c89feb69-b2cf-4886-9295-e1657179a88b-kube-api-access-g5kf5\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:16.072812 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:16.072773    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"]
Sep 17 19:55:16.075150 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:16.075119    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-54hlr"]
Sep 17 19:55:16.254963 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:16.254878    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e" path="/var/lib/kubelet/pods/7e7e7e1f-0c46-4fc1-a2df-3b95752d1b5e/volumes"
Sep 17 19:55:20.901718 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:20.901669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:20.908343 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:20.908319    2583 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:21.002738 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:21.002705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:21.002862 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:21.002821    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:21.005724 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:21.005705    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") DevicePath \"csi-1fe2660891d2e3d298cc42e9edc2df99890aec2342fcef00b3c8486894e6079f\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:21.291156 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:21.291126    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/aae21d65886ed100e34aa893258b7f849fafe11714576f6bec00075854c0ade3/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:21.304815 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:21.304789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"e2e-pvc-lora-download-j2cxd\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:21.311160 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:21.311142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:21.585622 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:21.585592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"]
Sep 17 19:55:21.588136 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:55:21.588109    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc89feb69_b2cf_4886_9295_e1657179a88b.slice/crio-092ad29a081c733ac51df0d2a3aaa5e4bcdab16fbf390a6cd8c99b8b06325393 WatchSource:0}: Error finding container 092ad29a081c733ac51df0d2a3aaa5e4bcdab16fbf390a6cd8c99b8b06325393: Status 404 returned error can't find the container with id 092ad29a081c733ac51df0d2a3aaa5e4bcdab16fbf390a6cd8c99b8b06325393
Sep 17 19:55:22.209744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:22.209681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd" event={"ID":"c89feb69-b2cf-4886-9295-e1657179a88b","Type":"ContainerStarted","Data":"2c522c5ad83896185d251951e31a30f81c10f04173c66938972b54f4a78afee4"}
Sep 17 19:55:22.209744 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:22.209749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd" event={"ID":"c89feb69-b2cf-4886-9295-e1657179a88b","Type":"ContainerStarted","Data":"092ad29a081c733ac51df0d2a3aaa5e4bcdab16fbf390a6cd8c99b8b06325393"}
Sep 17 19:55:22.228214 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:22.228163    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd" podStartSLOduration=11.228148922 podStartE2EDuration="11.228148922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:55:22.227285722 +0000 UTC m=+2924.275175676" watchObservedRunningTime="2026-09-17 19:55:22.228148922 +0000 UTC m=+2924.276028779"
Sep 17 19:55:24.222339 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:24.222300    2583 generic.go:356] "Generic (PLEG): container finished" podID="c89feb69-b2cf-4886-9295-e1657179a88b" containerID="2c522c5ad83896185d251951e31a30f81c10f04173c66938972b54f4a78afee4" exitCode=0
Sep 17 19:55:24.222759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:24.222375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd" event={"ID":"c89feb69-b2cf-4886-9295-e1657179a88b","Type":"ContainerDied","Data":"2c522c5ad83896185d251951e31a30f81c10f04173c66938972b54f4a78afee4"}
Sep 17 19:55:25.376653 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:25.376628    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:25.543062 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:25.542977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5kf5\" (UniqueName: \"kubernetes.io/projected/c89feb69-b2cf-4886-9295-e1657179a88b-kube-api-access-g5kf5\") pod \"c89feb69-b2cf-4886-9295-e1657179a88b\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") "
Sep 17 19:55:25.543207 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:25.543150    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"c89feb69-b2cf-4886-9295-e1657179a88b\" (UID: \"c89feb69-b2cf-4886-9295-e1657179a88b\") "
Sep 17 19:55:25.545477 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:25.545445    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89feb69-b2cf-4886-9295-e1657179a88b-kube-api-access-g5kf5" (OuterVolumeSpecName: "kube-api-access-g5kf5") pod "c89feb69-b2cf-4886-9295-e1657179a88b" (UID: "c89feb69-b2cf-4886-9295-e1657179a88b"). InnerVolumeSpecName "kube-api-access-g5kf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:25.546987 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:25.546962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67" (OuterVolumeSpecName: "model-storage") pod "c89feb69-b2cf-4886-9295-e1657179a88b" (UID: "c89feb69-b2cf-4886-9295-e1657179a88b"). InnerVolumeSpecName "pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:55:25.644253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:25.644221    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") on node \"ip-10-0-128-131.ec2.internal\" "
Sep 17 19:55:25.644253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:25.644252    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5kf5\" (UniqueName: \"kubernetes.io/projected/c89feb69-b2cf-4886-9295-e1657179a88b-kube-api-access-g5kf5\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:55:26.238586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:26.238547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd" event={"ID":"c89feb69-b2cf-4886-9295-e1657179a88b","Type":"ContainerDied","Data":"092ad29a081c733ac51df0d2a3aaa5e4bcdab16fbf390a6cd8c99b8b06325393"}
Sep 17 19:55:26.238586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:26.238585    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"
Sep 17 19:55:26.238586 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:26.238595    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092ad29a081c733ac51df0d2a3aaa5e4bcdab16fbf390a6cd8c99b8b06325393"
Sep 17 19:55:26.615938 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:26.615917    2583 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67") on node "ip-10-0-128-131.ec2.internal"
Sep 17 19:55:26.653090 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:26.653060    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"csi-1fe2660891d2e3d298cc42e9edc2df99890aec2342fcef00b3c8486894e6079f\""
Sep 17 19:55:42.358445 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.358413    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"]
Sep 17 19:55:42.358917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.358807    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89feb69-b2cf-4886-9295-e1657179a88b" containerName="storage-initializer"
Sep 17 19:55:42.358917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.358820    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89feb69-b2cf-4886-9295-e1657179a88b" containerName="storage-initializer"
Sep 17 19:55:42.358917 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.358884    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89feb69-b2cf-4886-9295-e1657179a88b" containerName="storage-initializer"
Sep 17 19:55:42.361878 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.361859    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.364759 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.364733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 19:55:42.364891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.364733    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 19:55:42.364891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.364737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-sbbt9\""
Sep 17 19:55:42.365018 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.364744    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 19:55:42.371306 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.371285    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"]
Sep 17 19:55:42.474043 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.474015    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.474170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.474053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.474170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.474084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stprw\" (UniqueName: \"kubernetes.io/projected/f4b144a7-b4ed-417c-bfa9-6d865f349947-kube-api-access-stprw\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.474170 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.474133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b144a7-b4ed-417c-bfa9-6d865f349947-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.474283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.474220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.474283 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.474269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.574823 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.574801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.574960 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.574852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stprw\" (UniqueName: \"kubernetes.io/projected/f4b144a7-b4ed-417c-bfa9-6d865f349947-kube-api-access-stprw\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.574960 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.574888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b144a7-b4ed-417c-bfa9-6d865f349947-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.574960 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.574917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.575131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.574971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.575131 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.575027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.575415 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.575395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.575495 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.575426    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.575495 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.575445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.577261 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.577229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.577442 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.577422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b144a7-b4ed-417c-bfa9-6d865f349947-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:42.583069 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:42.583050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stprw\" (UniqueName: \"kubernetes.io/projected/f4b144a7-b4ed-417c-bfa9-6d865f349947-kube-api-access-stprw\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.448605 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.448521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.454653 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.454628    2583 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.549771 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.549742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.549912 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.549893    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.553359 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.553334    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") DevicePath \"csi-1fe2660891d2e3d298cc42e9edc2df99890aec2342fcef00b3c8486894e6079f\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.638472 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.638446    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/aae21d65886ed100e34aa893258b7f849fafe11714576f6bec00075854c0ade3/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.659722 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.659668    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.673108 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.673088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:55:51.953341 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:51.953305    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"]
Sep 17 19:55:51.954743 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:55:51.954714    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b144a7_b4ed_417c_bfa9_6d865f349947.slice/crio-92666d3bc0d597832a8d3572a25dc75015a63a684b676109e427dd3b5c417406 WatchSource:0}: Error finding container 92666d3bc0d597832a8d3572a25dc75015a63a684b676109e427dd3b5c417406: Status 404 returned error can't find the container with id 92666d3bc0d597832a8d3572a25dc75015a63a684b676109e427dd3b5c417406
Sep 17 19:55:52.380102 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:52.380065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" event={"ID":"f4b144a7-b4ed-417c-bfa9-6d865f349947","Type":"ContainerStarted","Data":"c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58"}
Sep 17 19:55:52.380501 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:52.380117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" event={"ID":"f4b144a7-b4ed-417c-bfa9-6d865f349947","Type":"ContainerStarted","Data":"92666d3bc0d597832a8d3572a25dc75015a63a684b676109e427dd3b5c417406"}
Sep 17 19:55:52.399720 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:52.399646    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podStartSLOduration=10.399627519 podStartE2EDuration="10.399627519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:55:52.397611644 +0000 UTC m=+2954.445491505" watchObservedRunningTime="2026-09-17 19:55:52.399627519 +0000 UTC m=+2954.447507375"
Sep 17 19:55:52.673367 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:55:52.673288    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:56:01.673290 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:01.673250    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:56:01.674667 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:01.674638    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 17 19:56:11.674445 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:11.674396    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 17 19:56:21.673728 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:21.673670    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 17 19:56:31.674075 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:31.674025    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 17 19:56:41.674064 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:41.674019    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 17 19:56:42.280012 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.279982    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"]
Sep 17 19:56:42.283574 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.283558    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288082 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbs5x\" (UniqueName: \"kubernetes.io/projected/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kube-api-access-qbs5x\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288221 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288221 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288358 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288358 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288358 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288358 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-home\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288567 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.288631 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288610    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 19:56:42.288777 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288758    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 19:56:42.288858 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.288848    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:56:42.289386 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.289366    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-nmg8w\""
Sep 17 19:56:42.289490 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.289472    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 19:56:42.289555 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.289490    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 19:56:42.316341 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.316311    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"]
Sep 17 19:56:42.388977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.388939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.388977 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.388980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbs5x\" (UniqueName: \"kubernetes.io/projected/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kube-api-access-qbs5x\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389235 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-home\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389550 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389486    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389608 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389581    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389779 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-home\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389885 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389811    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.389953 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.389902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.391531 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.391508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.391907 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.391888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.397898 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.397870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbs5x\" (UniqueName: \"kubernetes.io/projected/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kube-api-access-qbs5x\") pod \"router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.595129 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.595088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:42.883021 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:42.882985    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"]
Sep 17 19:56:42.884723 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:56:42.884666    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f1fe00_fd88_4114_9b06_e7c6c7b4015e.slice/crio-20ab7fbfea9af3c48f9c3b5eb6e4a8d6d76410f96dd8dcb2a12fbdd5945d910f WatchSource:0}: Error finding container 20ab7fbfea9af3c48f9c3b5eb6e4a8d6d76410f96dd8dcb2a12fbdd5945d910f: Status 404 returned error can't find the container with id 20ab7fbfea9af3c48f9c3b5eb6e4a8d6d76410f96dd8dcb2a12fbdd5945d910f
Sep 17 19:56:43.687563 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:43.687519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" event={"ID":"59f1fe00-fd88-4114-9b06-e7c6c7b4015e","Type":"ContainerStarted","Data":"53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731"}
Sep 17 19:56:43.687766 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:43.687595    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" event={"ID":"59f1fe00-fd88-4114-9b06-e7c6c7b4015e","Type":"ContainerStarted","Data":"20ab7fbfea9af3c48f9c3b5eb6e4a8d6d76410f96dd8dcb2a12fbdd5945d910f"}
Sep 17 19:56:45.702423 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:45.702325    2583 generic.go:356] "Generic (PLEG): container finished" podID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerID="53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731" exitCode=0
Sep 17 19:56:45.702423 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:45.702379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" event={"ID":"59f1fe00-fd88-4114-9b06-e7c6c7b4015e","Type":"ContainerDied","Data":"53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731"}
Sep 17 19:56:46.711996 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:46.711963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" event={"ID":"59f1fe00-fd88-4114-9b06-e7c6c7b4015e","Type":"ContainerStarted","Data":"db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f"}
Sep 17 19:56:46.737339 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:46.737274    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podStartSLOduration=2.36640487 podStartE2EDuration="4.737257742s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.370852872s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:56:46.734266468 +0000 UTC m=+3008.782146324" watchObservedRunningTime="2026-09-17 19:56:46.737257742 +0000 UTC m=+3008.785137598"
Sep 17 19:56:51.674506 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:51.674453    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 17 19:56:52.596111 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:52.596074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:52.596295 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:52.596163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:56:52.597758 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:56:52.597728    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:57:01.674525 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:01.674474    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Sep 17 19:57:02.596414 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:02.596371    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:57:11.683427 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:11.683394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:57:11.691315 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:11.691286    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:57:12.595516 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:12.595472    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:57:22.595677 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:22.595573    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:57:32.596315 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:32.596267    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:57:38.967506 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:38.967469    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb_f4b144a7-b4ed-417c-bfa9-6d865f349947/main/0.log"
Sep 17 19:57:39.968930 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:39.968893    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"]
Sep 17 19:57:42.596471 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:42.596422    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:57:52.596146 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:52.596106    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:57:54.995255 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:54.995207    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main" containerID="cri-o://c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58" gracePeriod=45
Sep 17 19:57:55.652700 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.652655    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:57:55.726251 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-model-cache\") pod \"f4b144a7-b4ed-417c-bfa9-6d865f349947\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") "
Sep 17 19:57:55.726251 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726222    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-tmp-dir\") pod \"f4b144a7-b4ed-417c-bfa9-6d865f349947\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") "
Sep 17 19:57:55.726428 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726281    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-dshm\") pod \"f4b144a7-b4ed-417c-bfa9-6d865f349947\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") "
Sep 17 19:57:55.726428 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726392    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") pod \"f4b144a7-b4ed-417c-bfa9-6d865f349947\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") "
Sep 17 19:57:55.726531 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726432    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4b144a7-b4ed-417c-bfa9-6d865f349947" (UID: "f4b144a7-b4ed-417c-bfa9-6d865f349947"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:57:55.726531 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726441    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b144a7-b4ed-417c-bfa9-6d865f349947-tls-certs\") pod \"f4b144a7-b4ed-417c-bfa9-6d865f349947\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") "
Sep 17 19:57:55.726531 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stprw\" (UniqueName: \"kubernetes.io/projected/f4b144a7-b4ed-417c-bfa9-6d865f349947-kube-api-access-stprw\") pod \"f4b144a7-b4ed-417c-bfa9-6d865f349947\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") "
Sep 17 19:57:55.726723 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726540    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-home\") pod \"f4b144a7-b4ed-417c-bfa9-6d865f349947\" (UID: \"f4b144a7-b4ed-417c-bfa9-6d865f349947\") "
Sep 17 19:57:55.726954 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.726934    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:57:55.727196 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.727014    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-home" (OuterVolumeSpecName: "home") pod "f4b144a7-b4ed-417c-bfa9-6d865f349947" (UID: "f4b144a7-b4ed-417c-bfa9-6d865f349947"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:57:55.728870 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.728842    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b144a7-b4ed-417c-bfa9-6d865f349947-kube-api-access-stprw" (OuterVolumeSpecName: "kube-api-access-stprw") pod "f4b144a7-b4ed-417c-bfa9-6d865f349947" (UID: "f4b144a7-b4ed-417c-bfa9-6d865f349947"). InnerVolumeSpecName "kube-api-access-stprw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:57:55.729095 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.729070    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b144a7-b4ed-417c-bfa9-6d865f349947-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4b144a7-b4ed-417c-bfa9-6d865f349947" (UID: "f4b144a7-b4ed-417c-bfa9-6d865f349947"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:57:55.729865 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.729845    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-dshm" (OuterVolumeSpecName: "dshm") pod "f4b144a7-b4ed-417c-bfa9-6d865f349947" (UID: "f4b144a7-b4ed-417c-bfa9-6d865f349947"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:57:55.734989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.734972    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67" (OuterVolumeSpecName: "kserve-pvc-source") pod "f4b144a7-b4ed-417c-bfa9-6d865f349947" (UID: "f4b144a7-b4ed-417c-bfa9-6d865f349947"). InnerVolumeSpecName "pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:57:55.743703 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.743664    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4b144a7-b4ed-417c-bfa9-6d865f349947" (UID: "f4b144a7-b4ed-417c-bfa9-6d865f349947"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:57:55.827796 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.827758    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:57:55.827908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.827813    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") on node \"ip-10-0-128-131.ec2.internal\" "
Sep 17 19:57:55.827908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.827825    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b144a7-b4ed-417c-bfa9-6d865f349947-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:57:55.827908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.827835    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stprw\" (UniqueName: \"kubernetes.io/projected/f4b144a7-b4ed-417c-bfa9-6d865f349947-kube-api-access-stprw\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:57:55.827908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.827843    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:57:55.827908 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.827851    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b144a7-b4ed-417c-bfa9-6d865f349947-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:57:55.855749 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.855731    2583 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67") on node "ip-10-0-128-131.ec2.internal"
Sep 17 19:57:55.928968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:55.928935    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-b2ff1a1d-abad-49fe-93a4-4424fd396e7d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0555273d3a2c1ac67\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"csi-1fe2660891d2e3d298cc42e9edc2df99890aec2342fcef00b3c8486894e6079f\""
Sep 17 19:57:56.164909 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.164865    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerID="c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58" exitCode=0
Sep 17 19:57:56.165443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.164960    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"
Sep 17 19:57:56.165443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.164977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" event={"ID":"f4b144a7-b4ed-417c-bfa9-6d865f349947","Type":"ContainerDied","Data":"c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58"}
Sep 17 19:57:56.165443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.165138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb" event={"ID":"f4b144a7-b4ed-417c-bfa9-6d865f349947","Type":"ContainerDied","Data":"92666d3bc0d597832a8d3572a25dc75015a63a684b676109e427dd3b5c417406"}
Sep 17 19:57:56.165443 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.165173    2583 scope.go:117] "RemoveContainer" containerID="c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58"
Sep 17 19:57:56.179609 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.179576    2583 scope.go:117] "RemoveContainer" containerID="c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58"
Sep 17 19:57:56.180036 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:57:56.180007    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58\": container with ID starting with c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58 not found: ID does not exist" containerID="c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58"
Sep 17 19:57:56.180126 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.180049    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58"} err="failed to get container status \"c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58\": rpc error: code = NotFound desc = could not find container \"c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58\": container with ID starting with c7cd9a829952281166d60bd18ad3ba17983b32e8a1dd8dfc8b4d330d76dbfa58 not found: ID does not exist"
Sep 17 19:57:56.196051 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.196019    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"]
Sep 17 19:57:56.202357 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.202300    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-7jmlb"]
Sep 17 19:57:56.259848 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:57:56.259808    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" path="/var/lib/kubelet/pods/f4b144a7-b4ed-417c-bfa9-6d865f349947/volumes"
Sep 17 19:58:02.596185 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:02.596142    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:58:05.078973 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:05.078934    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"]
Sep 17 19:58:05.082446 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:05.082419    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-j2cxd"]
Sep 17 19:58:06.255337 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:06.255303    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89feb69-b2cf-4886-9295-e1657179a88b" path="/var/lib/kubelet/pods/c89feb69-b2cf-4886-9295-e1657179a88b/volumes"
Sep 17 19:58:07.555905 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.553404    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"]
Sep 17 19:58:07.555905 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.554331    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main"
Sep 17 19:58:07.555905 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.554353    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main"
Sep 17 19:58:07.555905 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.554546    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b144a7-b4ed-417c-bfa9-6d865f349947" containerName="main"
Sep 17 19:58:07.565762 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.565667    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.567763 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.567736    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"]
Sep 17 19:58:07.568273 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.568249    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 19:58:07.568621 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.568593    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 19:58:07.568779 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.568756    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 19:58:07.568885 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.568863    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 19:58:07.568968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.568908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-m242m\""
Sep 17 19:58:07.568968 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.568866    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:58:07.621646 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-dshm\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.621792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tls-certs\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.621792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-model-cache\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.621792 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tmp-dir\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.621900 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kserve-provision-location\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.621900 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea6319-60d1-4235-a9f5-ef7ac3119b44-cabundle-cert\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.621900 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k78w\" (UniqueName: \"kubernetes.io/projected/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kube-api-access-6k78w\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.621999 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.621904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-home\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722541 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-home\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-dshm\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tls-certs\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-model-cache\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722655 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tmp-dir\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kserve-provision-location\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea6319-60d1-4235-a9f5-ef7ac3119b44-cabundle-cert\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.722891 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k78w\" (UniqueName: \"kubernetes.io/projected/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kube-api-access-6k78w\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.723045 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.722979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-home\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.723099 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.723044    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tmp-dir\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.723221 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.723200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kserve-provision-location\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.723486 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.723465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-model-cache\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.723850 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.723830    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea6319-60d1-4235-a9f5-ef7ac3119b44-cabundle-cert\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.725216 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.725188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-dshm\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.725475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.725455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tls-certs\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.731934 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.731910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k78w\" (UniqueName: \"kubernetes.io/projected/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kube-api-access-6k78w\") pod \"tls-verification-test-kserve-c766b4dfc-l6gjb\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:07.878220 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:07.878126    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:08.162358 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:08.162327    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"]
Sep 17 19:58:08.164753 ip-10-0-128-131 kubenswrapper[2583]: W0917 19:58:08.164723    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ea6319_60d1_4235_a9f5_ef7ac3119b44.slice/crio-748aa49578d123d78479d0a57e56f7607b37bbc8dbbf95b97d336a5b7f2f1d66 WatchSource:0}: Error finding container 748aa49578d123d78479d0a57e56f7607b37bbc8dbbf95b97d336a5b7f2f1d66: Status 404 returned error can't find the container with id 748aa49578d123d78479d0a57e56f7607b37bbc8dbbf95b97d336a5b7f2f1d66
Sep 17 19:58:08.242470 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:08.242432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" event={"ID":"31ea6319-60d1-4235-a9f5-ef7ac3119b44","Type":"ContainerStarted","Data":"361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182"}
Sep 17 19:58:08.242601 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:08.242502    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" event={"ID":"31ea6319-60d1-4235-a9f5-ef7ac3119b44","Type":"ContainerStarted","Data":"748aa49578d123d78479d0a57e56f7607b37bbc8dbbf95b97d336a5b7f2f1d66"}
Sep 17 19:58:11.262812 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:11.262779    2583 generic.go:356] "Generic (PLEG): container finished" podID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerID="361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182" exitCode=0
Sep 17 19:58:11.263216 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:11.262854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" event={"ID":"31ea6319-60d1-4235-a9f5-ef7ac3119b44","Type":"ContainerDied","Data":"361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182"}
Sep 17 19:58:12.272391 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:12.272340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" event={"ID":"31ea6319-60d1-4235-a9f5-ef7ac3119b44","Type":"ContainerStarted","Data":"9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213"}
Sep 17 19:58:12.296179 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:12.296112    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podStartSLOduration=2.925076371 podStartE2EDuration="5.296086914s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.371010543s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:58:12.295039538 +0000 UTC m=+3094.342919408" watchObservedRunningTime="2026-09-17 19:58:12.296086914 +0000 UTC m=+3094.343966791"
Sep 17 19:58:12.596233 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:12.596188    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:58:12.879336 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:12.879229    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:17.878572 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:17.878535    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 19:58:17.879805 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:17.879776    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:58:22.595633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:22.595585    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:58:27.879511 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:27.879457    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:58:32.595754 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:32.595674    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Sep 17 19:58:37.879276 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:37.879227    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:58:42.606253 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:42.606221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:58:42.614225 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:42.614200    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:58:46.830425 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:46.830391    2583 scope.go:117] "RemoveContainer" containerID="57ae05628c9e367c69279f9f49002de1f6a5c85568aea9370f65fb34628facdb"
Sep 17 19:58:47.878811 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:47.878763    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:58:57.879021 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:58:57.878933    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:59:01.655732 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:01.655673    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"]
Sep 17 19:59:07.878939 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:07.878890    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:59:16.679108 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:16.679056    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main" containerID="cri-o://db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f" gracePeriod=45
Sep 17 19:59:17.428127 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.428103    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:59:17.561728 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-cabundle-cert\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.561897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kserve-provision-location\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.561897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-home\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.561897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561800    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-model-cache\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.561897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561823    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tls-certs\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.561897 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-dshm\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.562153 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbs5x\" (UniqueName: \"kubernetes.io/projected/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kube-api-access-qbs5x\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.562153 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.561972    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tmp-dir\") pod \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\" (UID: \"59f1fe00-fd88-4114-9b06-e7c6c7b4015e\") "
Sep 17 19:59:17.562153 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.562083    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:59:17.562153 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.562094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-model-cache" (OuterVolumeSpecName: "model-cache") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.562411 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.562276    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.562475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.562419    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.562535 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.562507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-home" (OuterVolumeSpecName: "home") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.564633 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.564603    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kube-api-access-qbs5x" (OuterVolumeSpecName: "kube-api-access-qbs5x") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "kube-api-access-qbs5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:59:17.564980 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.564958    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-dshm" (OuterVolumeSpecName: "dshm") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.565125 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.565101    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:59:17.576101 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.576068    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.586948 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.586926    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59f1fe00-fd88-4114-9b06-e7c6c7b4015e" (UID: "59f1fe00-fd88-4114-9b06-e7c6c7b4015e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:59:17.662989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.662963    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.662989 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.662986    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.663122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.662995    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.663122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.663003    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.663122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.663011    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbs5x\" (UniqueName: \"kubernetes.io/projected/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-kube-api-access-qbs5x\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.663122 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.663019    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f1fe00-fd88-4114-9b06-e7c6c7b4015e-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 19:59:17.691475 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.691449    2583 generic.go:356] "Generic (PLEG): container finished" podID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerID="db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f" exitCode=0
Sep 17 19:59:17.691801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.691515    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"
Sep 17 19:59:17.691801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.691536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" event={"ID":"59f1fe00-fd88-4114-9b06-e7c6c7b4015e","Type":"ContainerDied","Data":"db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f"}
Sep 17 19:59:17.691801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.691598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf" event={"ID":"59f1fe00-fd88-4114-9b06-e7c6c7b4015e","Type":"ContainerDied","Data":"20ab7fbfea9af3c48f9c3b5eb6e4a8d6d76410f96dd8dcb2a12fbdd5945d910f"}
Sep 17 19:59:17.691801 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.691626    2583 scope.go:117] "RemoveContainer" containerID="db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f"
Sep 17 19:59:17.700727 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.700710    2583 scope.go:117] "RemoveContainer" containerID="53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731"
Sep 17 19:59:17.709109 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.709090    2583 scope.go:117] "RemoveContainer" containerID="db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f"
Sep 17 19:59:17.709375 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:59:17.709356    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f\": container with ID starting with db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f not found: ID does not exist" containerID="db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f"
Sep 17 19:59:17.709451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.709384    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f"} err="failed to get container status \"db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f\": rpc error: code = NotFound desc = could not find container \"db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f\": container with ID starting with db77efb7f9d14f4d43014ec816f3f4cbc0f668ed5bf6063d76d3c6746b97951f not found: ID does not exist"
Sep 17 19:59:17.709451 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.709403    2583 scope.go:117] "RemoveContainer" containerID="53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731"
Sep 17 19:59:17.709675 ip-10-0-128-131 kubenswrapper[2583]: E0917 19:59:17.709652    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731\": container with ID starting with 53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731 not found: ID does not exist" containerID="53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731"
Sep 17 19:59:17.709786 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.709681    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731"} err="failed to get container status \"53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731\": rpc error: code = NotFound desc = could not find container \"53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731\": container with ID starting with 53cbb1ef2cc653fdb89c95efcec782819db4eb64b341d441c36a1dad4600b731 not found: ID does not exist"
Sep 17 19:59:17.718173 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.718138    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"]
Sep 17 19:59:17.725129 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.725107    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f48f6fdc4-l95kf"]
Sep 17 19:59:17.879101 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:17.879011    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:59:18.255518 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:18.255433    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" path="/var/lib/kubelet/pods/59f1fe00-fd88-4114-9b06-e7c6c7b4015e/volumes"
Sep 17 19:59:27.879009 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:27.878969    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:59:37.879063 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:37.879011    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:59:47.878903 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:47.878850    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 19:59:57.879051 ip-10-0-128-131 kubenswrapper[2583]: I0917 19:59:57.878997    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 20:00:07.889093 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:07.889059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 20:00:07.896659 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:07.896635    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 20:00:13.592998 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:13.592958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"]
Sep 17 20:00:28.617108 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:28.617060    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" containerID="cri-o://9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213" gracePeriod=45
Sep 17 20:00:28.890503 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:28.890419    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Sep 17 20:00:29.366496 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.366472    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 20:00:29.451653 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451594    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-model-cache\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.451653 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-dshm\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.451842 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451669    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tmp-dir\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.451842 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451818    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea6319-60d1-4235-a9f5-ef7ac3119b44-cabundle-cert\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.451947 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451853    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-home\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.451947 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kserve-provision-location\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.451947 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451930    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tls-certs\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.452118 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-model-cache" (OuterVolumeSpecName: "model-cache") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:29.452118 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.451996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k78w\" (UniqueName: \"kubernetes.io/projected/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kube-api-access-6k78w\") pod \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\" (UID: \"31ea6319-60d1-4235-a9f5-ef7ac3119b44\") "
Sep 17 20:00:29.452257 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.452228    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ea6319-60d1-4235-a9f5-ef7ac3119b44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:00:29.452316 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.452264    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:29.452669 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.452604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-home" (OuterVolumeSpecName: "home") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:29.454307 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.454278    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:00:29.454415 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.454341    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-dshm" (OuterVolumeSpecName: "dshm") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:29.454733 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.454668    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kube-api-access-6k78w" (OuterVolumeSpecName: "kube-api-access-6k78w") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "kube-api-access-6k78w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:00:29.465639 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.465617    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:29.476553 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.476531    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31ea6319-60d1-4235-a9f5-ef7ac3119b44" (UID: "31ea6319-60d1-4235-a9f5-ef7ac3119b44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:00:29.552899 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.552877    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k78w\" (UniqueName: \"kubernetes.io/projected/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kube-api-access-6k78w\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:29.552899 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.552898    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:29.553021 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.552909    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:29.553021 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.552918    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ea6319-60d1-4235-a9f5-ef7ac3119b44-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:29.553021 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.552925    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:29.553021 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.552933    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ea6319-60d1-4235-a9f5-ef7ac3119b44-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:29.553021 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:29.552940    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ea6319-60d1-4235-a9f5-ef7ac3119b44-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:00:30.125699 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.125659    2583 generic.go:356] "Generic (PLEG): container finished" podID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerID="9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213" exitCode=0
Sep 17 20:00:30.126195 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.125734    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"
Sep 17 20:00:30.126195 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.125744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" event={"ID":"31ea6319-60d1-4235-a9f5-ef7ac3119b44","Type":"ContainerDied","Data":"9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213"}
Sep 17 20:00:30.126195 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.125834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb" event={"ID":"31ea6319-60d1-4235-a9f5-ef7ac3119b44","Type":"ContainerDied","Data":"748aa49578d123d78479d0a57e56f7607b37bbc8dbbf95b97d336a5b7f2f1d66"}
Sep 17 20:00:30.126195 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.125857    2583 scope.go:117] "RemoveContainer" containerID="9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213"
Sep 17 20:00:30.139199 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.139183    2583 scope.go:117] "RemoveContainer" containerID="361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182"
Sep 17 20:00:30.147788 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.147754    2583 scope.go:117] "RemoveContainer" containerID="9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213"
Sep 17 20:00:30.148191 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:00:30.148089    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213\": container with ID starting with 9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213 not found: ID does not exist" containerID="9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213"
Sep 17 20:00:30.148191 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.148169    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213"} err="failed to get container status \"9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213\": rpc error: code = NotFound desc = could not find container \"9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213\": container with ID starting with 9f31f3e49481bdfeab69c0a13f12916200a1f581e389f545714891d6e9502213 not found: ID does not exist"
Sep 17 20:00:30.148191 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.148189    2583 scope.go:117] "RemoveContainer" containerID="361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182"
Sep 17 20:00:30.148509 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:00:30.148476    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182\": container with ID starting with 361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182 not found: ID does not exist" containerID="361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182"
Sep 17 20:00:30.148596 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.148508    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182"} err="failed to get container status \"361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182\": rpc error: code = NotFound desc = could not find container \"361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182\": container with ID starting with 361054ed2acd23187f271680bd278f40529d0c6f46fe2d51614ca22542ad6182 not found: ID does not exist"
Sep 17 20:00:30.151875 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.151851    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"]
Sep 17 20:00:30.156823 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.156798    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-c766b4dfc-l6gjb"]
Sep 17 20:00:30.255355 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:00:30.255320    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" path="/var/lib/kubelet/pods/31ea6319-60d1-4235-a9f5-ef7ac3119b44/volumes"
Sep 17 20:01:05.883475 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.883437    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"]
Sep 17 20:01:05.884044 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884024    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="storage-initializer"
Sep 17 20:01:05.884099 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884047    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="storage-initializer"
Sep 17 20:01:05.884099 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884062    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main"
Sep 17 20:01:05.884099 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884068    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main"
Sep 17 20:01:05.884099 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884076    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main"
Sep 17 20:01:05.884099 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884083    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main"
Sep 17 20:01:05.884099 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884099    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="storage-initializer"
Sep 17 20:01:05.884281 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884106    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="storage-initializer"
Sep 17 20:01:05.884281 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884154    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f1fe00-fd88-4114-9b06-e7c6c7b4015e" containerName="main"
Sep 17 20:01:05.884281 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.884164    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ea6319-60d1-4235-a9f5-ef7ac3119b44" containerName="main"
Sep 17 20:01:05.888461 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.888445    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.891263 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.891239    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 20:01:05.891263 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.891256    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 20:01:05.891486 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.891302    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-x2xpm\""
Sep 17 20:01:05.891486 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.891440    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 20:01:05.891486 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.891448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 20:01:05.891486 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.891440    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 20:01:05.898959 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.898938    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"]
Sep 17 20:01:05.935920 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.935894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-dshm\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.936045 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.935929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tmp-dir\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.936045 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.935989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-home\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.936172 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.936037    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tls-certs\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.936172 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.936079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a564c6-7e13-4fd1-b766-4c1b477f74dc-cabundle-cert\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.936172 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.936111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv5k\" (UniqueName: \"kubernetes.io/projected/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kube-api-access-4zv5k\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.936172 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.936140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kserve-provision-location\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:05.936409 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:05.936261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-model-cache\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037165 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tls-certs\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037267 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a564c6-7e13-4fd1-b766-4c1b477f74dc-cabundle-cert\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037267 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037188    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv5k\" (UniqueName: \"kubernetes.io/projected/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kube-api-access-4zv5k\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037267 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kserve-provision-location\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037267 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-model-cache\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037439 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-dshm\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037439 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tmp-dir\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037439 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-home\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037810 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-home\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037924 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kserve-provision-location\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.037996 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.037938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tmp-dir\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.038109 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.038043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a564c6-7e13-4fd1-b766-4c1b477f74dc-cabundle-cert\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.038175 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.038152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-model-cache\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.039735 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.039716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-dshm\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.040024 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.040005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tls-certs\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.044760 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.044741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv5k\" (UniqueName: \"kubernetes.io/projected/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kube-api-access-4zv5k\") pod \"prestop-hook-test-kserve-67b9d98997-h5sbt\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.199939 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.199841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:06.483415 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:06.483384    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"]
Sep 17 20:01:06.484962 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:01:06.484934    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a564c6_7e13_4fd1_b766_4c1b477f74dc.slice/crio-08038d2ed3d9c56edb0ed2d674218b118d484b0dc92ff0003de45cf962c0bdd5 WatchSource:0}: Error finding container 08038d2ed3d9c56edb0ed2d674218b118d484b0dc92ff0003de45cf962c0bdd5: Status 404 returned error can't find the container with id 08038d2ed3d9c56edb0ed2d674218b118d484b0dc92ff0003de45cf962c0bdd5
Sep 17 20:01:07.351022 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:07.350979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" event={"ID":"73a564c6-7e13-4fd1-b766-4c1b477f74dc","Type":"ContainerStarted","Data":"8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d"}
Sep 17 20:01:07.351435 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:07.351046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" event={"ID":"73a564c6-7e13-4fd1-b766-4c1b477f74dc","Type":"ContainerStarted","Data":"08038d2ed3d9c56edb0ed2d674218b118d484b0dc92ff0003de45cf962c0bdd5"}
Sep 17 20:01:09.366731 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:09.366682    2583 generic.go:356] "Generic (PLEG): container finished" podID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerID="8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d" exitCode=0
Sep 17 20:01:09.366731 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:09.366714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" event={"ID":"73a564c6-7e13-4fd1-b766-4c1b477f74dc","Type":"ContainerDied","Data":"8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d"}
Sep 17 20:01:10.375554 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:10.375522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" event={"ID":"73a564c6-7e13-4fd1-b766-4c1b477f74dc","Type":"ContainerStarted","Data":"9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f"}
Sep 17 20:01:10.406977 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:10.406924    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podStartSLOduration=3.154543286 podStartE2EDuration="5.406910641s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.252367355s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:01:10.404000978 +0000 UTC m=+3272.451880856" watchObservedRunningTime="2026-09-17 20:01:10.406910641 +0000 UTC m=+3272.454790498"
Sep 17 20:01:11.200491 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:11.200444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:16.200296 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:16.200255    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:01:16.201972 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:16.201938    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:01:26.200447 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:26.200402    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:01:36.200698 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:36.200653    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:01:46.201066 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:46.201022    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:01:46.888193 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:46.888162    2583 scope.go:117] "RemoveContainer" containerID="2c522c5ad83896185d251951e31a30f81c10f04173c66938972b54f4a78afee4"
Sep 17 20:01:56.200551 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:01:56.200437    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:02:06.200412 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:06.200359    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:02:16.200505 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:16.200459    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:02:26.201288 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:26.201240    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:02:36.200320 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:36.200273    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:02:46.200962 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:46.200923    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:02:50.301218 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.301168    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 20:02:50.339852 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.339811    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 20:02:50.340046 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.339932    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.342505 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.342478    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 20:02:50.342505 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.342494    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 20:02:50.342505 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.342491    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 20:02:50.342933 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.342916    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-9f8gb\""
Sep 17 20:02:50.343629 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.343612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rhnr7\""
Sep 17 20:02:50.343762 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.343618    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 20:02:50.343762 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.343736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 20:02:50.447871 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.447824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.447871 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.447862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5bp\" (UniqueName: \"kubernetes.io/projected/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kube-api-access-ml5bp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.448141 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.447891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.448141 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.447985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.448141 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.448036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.448141 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.448089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.448141 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.448111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.448141 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.448128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549309 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549309 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549576 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549576 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5bp\" (UniqueName: \"kubernetes.io/projected/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kube-api-access-ml5bp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549576 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549576 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549576 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549576 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549978 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549978 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549978 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.549978 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.549976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.550221 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.550055    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.552287 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.552231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.552394 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.552372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.558389 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.558366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5bp\" (UniqueName: \"kubernetes.io/projected/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kube-api-access-ml5bp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.651153 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.651122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:50.972414 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:50.972388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 20:02:50.974987 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:02:50.974950    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd73bff5_3fcc_446f_bb58_7e9052c0f24d.slice/crio-ffd36f1796451f34581f3ecd4f806ff6de89ceb720173581fd7e980d9c088e90 WatchSource:0}: Error finding container ffd36f1796451f34581f3ecd4f806ff6de89ceb720173581fd7e980d9c088e90: Status 404 returned error can't find the container with id ffd36f1796451f34581f3ecd4f806ff6de89ceb720173581fd7e980d9c088e90
Sep 17 20:02:51.977543 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:51.977499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cd73bff5-3fcc-446f-bb58-7e9052c0f24d","Type":"ContainerStarted","Data":"66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1"}
Sep 17 20:02:51.978115 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:51.977561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cd73bff5-3fcc-446f-bb58-7e9052c0f24d","Type":"ContainerStarted","Data":"ffd36f1796451f34581f3ecd4f806ff6de89ceb720173581fd7e980d9c088e90"}
Sep 17 20:02:53.991048 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:53.990965    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerID="66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1" exitCode=0
Sep 17 20:02:53.991396 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:53.991041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cd73bff5-3fcc-446f-bb58-7e9052c0f24d","Type":"ContainerDied","Data":"66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1"}
Sep 17 20:02:55.003998 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:55.003957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cd73bff5-3fcc-446f-bb58-7e9052c0f24d","Type":"ContainerStarted","Data":"1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc"}
Sep 17 20:02:55.029394 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:55.029329    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.5160554729999998 podStartE2EDuration="5.029313351s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.513257878s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:02:55.02642978 +0000 UTC m=+3377.074309663" watchObservedRunningTime="2026-09-17 20:02:55.029313351 +0000 UTC m=+3377.077193208"
Sep 17 20:02:55.651762 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:55.651721    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:02:56.200315 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:02:56.200269    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 17 20:03:00.652081 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:00.652045    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:03:00.653414 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:00.653380    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:03:06.212095 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:06.212040    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:03:06.221082 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:06.221049    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:03:09.323628 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.323589    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"]
Sep 17 20:03:09.347058 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.347019    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"]
Sep 17 20:03:09.350046 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.350023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.362739 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.362717    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"]
Sep 17 20:03:09.523561 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxxm\" (UniqueName: \"kubernetes.io/projected/2068d01c-da4d-465c-ace8-840295a2ec68-kube-api-access-nxxxm\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.523758 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-home\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.523758 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-model-cache\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.523758 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-tmp-dir\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.523758 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-dshm\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.523758 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-kserve-provision-location\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.523758 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2068d01c-da4d-465c-ace8-840295a2ec68-tls-certs\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.523970 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.523765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2068d01c-da4d-465c-ace8-840295a2ec68-cabundle-cert\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.624835 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxxm\" (UniqueName: \"kubernetes.io/projected/2068d01c-da4d-465c-ace8-840295a2ec68-kube-api-access-nxxxm\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.624835 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-home\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.624835 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-model-cache\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625105 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-tmp-dir\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625105 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-dshm\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625105 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-kserve-provision-location\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625105 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2068d01c-da4d-465c-ace8-840295a2ec68-tls-certs\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625105 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.624944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2068d01c-da4d-465c-ace8-840295a2ec68-cabundle-cert\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625371 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.625197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-home\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625371 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.625298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-model-cache\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625635 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.625600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2068d01c-da4d-465c-ace8-840295a2ec68-cabundle-cert\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625957 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.625801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-kserve-provision-location\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.625957 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.625809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-tmp-dir\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.627749 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.627721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-dshm\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.627898 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.627879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2068d01c-da4d-465c-ace8-840295a2ec68-tls-certs\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.633238 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.633210    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxxm\" (UniqueName: \"kubernetes.io/projected/2068d01c-da4d-465c-ace8-840295a2ec68-kube-api-access-nxxxm\") pod \"prestop-hook-test-kserve-67b9d98997-2qkj6\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.660645 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.660619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:09.939359 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:03:09.939312    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2068d01c_da4d_465c_ace8_840295a2ec68.slice/crio-ac14a5cb838f591c3eb0149822f0419396cc1f768f3910a270fe8f24ce7b8dcc WatchSource:0}: Error finding container ac14a5cb838f591c3eb0149822f0419396cc1f768f3910a270fe8f24ce7b8dcc: Status 404 returned error can't find the container with id ac14a5cb838f591c3eb0149822f0419396cc1f768f3910a270fe8f24ce7b8dcc
Sep 17 20:03:09.940544 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:09.940517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"]
Sep 17 20:03:10.094701 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:10.094658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" event={"ID":"2068d01c-da4d-465c-ace8-840295a2ec68","Type":"ContainerStarted","Data":"dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b"}
Sep 17 20:03:10.094859 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:10.094743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" event={"ID":"2068d01c-da4d-465c-ace8-840295a2ec68","Type":"ContainerStarted","Data":"ac14a5cb838f591c3eb0149822f0419396cc1f768f3910a270fe8f24ce7b8dcc"}
Sep 17 20:03:10.652134 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:10.652096    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:03:13.116670 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:13.116633    2583 generic.go:356] "Generic (PLEG): container finished" podID="2068d01c-da4d-465c-ace8-840295a2ec68" containerID="dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b" exitCode=0
Sep 17 20:03:13.117074 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:13.116725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" event={"ID":"2068d01c-da4d-465c-ace8-840295a2ec68","Type":"ContainerDied","Data":"dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b"}
Sep 17 20:03:14.128988 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:14.128947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" event={"ID":"2068d01c-da4d-465c-ace8-840295a2ec68","Type":"ContainerStarted","Data":"369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd"}
Sep 17 20:03:14.151574 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:14.151513    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" podStartSLOduration=2.838363495 podStartE2EDuration="5.151492206s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.313128711s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:03:14.149880335 +0000 UTC m=+3396.197760193" watchObservedRunningTime="2026-09-17 20:03:14.151492206 +0000 UTC m=+3396.199372075"
Sep 17 20:03:14.661031 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:14.660997    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:19.661589 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:19.661489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:03:19.663346 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:19.663316    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": dial tcp 10.134.0.85:8000: connect: connection refused"
Sep 17 20:03:20.652536 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:20.652482    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:03:24.348796 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:24.348741    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main" containerID="cri-o://9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f" gracePeriod=45
Sep 17 20:03:25.106709 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.106668    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:03:25.176016 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.175903    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kserve-provision-location\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176016 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.175957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a564c6-7e13-4fd1-b766-4c1b477f74dc-cabundle-cert\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176016 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.175997    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tmp-dir\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176317 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.176052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zv5k\" (UniqueName: \"kubernetes.io/projected/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kube-api-access-4zv5k\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176317 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.176095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tls-certs\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176317 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.176120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-model-cache\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176317 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.176193    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-dshm\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176317 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.176212    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-home\") pod \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\" (UID: \"73a564c6-7e13-4fd1-b766-4c1b477f74dc\") "
Sep 17 20:03:25.176959 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.176932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-home" (OuterVolumeSpecName: "home") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:03:25.177100 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.176980    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a564c6-7e13-4fd1-b766-4c1b477f74dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:03:25.178246 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.178222    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:03:25.181151 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.181111    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kube-api-access-4zv5k" (OuterVolumeSpecName: "kube-api-access-4zv5k") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "kube-api-access-4zv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:03:25.181397 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.181354    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-dshm" (OuterVolumeSpecName: "dshm") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:03:25.182872 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.182842    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:03:25.193530 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.193505    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:03:25.209168 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.209136    2583 generic.go:356] "Generic (PLEG): container finished" podID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerID="9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f" exitCode=0
Sep 17 20:03:25.209284 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.209208    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"
Sep 17 20:03:25.209284 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.209213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" event={"ID":"73a564c6-7e13-4fd1-b766-4c1b477f74dc","Type":"ContainerDied","Data":"9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f"}
Sep 17 20:03:25.209406 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.209319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt" event={"ID":"73a564c6-7e13-4fd1-b766-4c1b477f74dc","Type":"ContainerDied","Data":"08038d2ed3d9c56edb0ed2d674218b118d484b0dc92ff0003de45cf962c0bdd5"}
Sep 17 20:03:25.209406 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.209355    2583 scope.go:117] "RemoveContainer" containerID="9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f"
Sep 17 20:03:25.216496 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.216460    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73a564c6-7e13-4fd1-b766-4c1b477f74dc" (UID: "73a564c6-7e13-4fd1-b766-4c1b477f74dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:03:25.219707 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.219667    2583 scope.go:117] "RemoveContainer" containerID="8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d"
Sep 17 20:03:25.229134 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.229116    2583 scope.go:117] "RemoveContainer" containerID="9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f"
Sep 17 20:03:25.229442 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:03:25.229417    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f\": container with ID starting with 9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f not found: ID does not exist" containerID="9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f"
Sep 17 20:03:25.229511 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.229455    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f"} err="failed to get container status \"9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f\": rpc error: code = NotFound desc = could not find container \"9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f\": container with ID starting with 9096331d6a8561a9d599d43188f0144739c131cf06eb22519688f38ae8559e7f not found: ID does not exist"
Sep 17 20:03:25.229511 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.229481    2583 scope.go:117] "RemoveContainer" containerID="8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d"
Sep 17 20:03:25.229799 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:03:25.229776    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d\": container with ID starting with 8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d not found: ID does not exist" containerID="8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d"
Sep 17 20:03:25.229862 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.229807    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d"} err="failed to get container status \"8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d\": rpc error: code = NotFound desc = could not find container \"8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d\": container with ID starting with 8b10c0bb6034268c726206cc8305274c24ea7c626c2e8c4815c38864f5f9a54d not found: ID does not exist"
Sep 17 20:03:25.276886 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276857    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.276886 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276883    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.277069 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276895    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.277069 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276905    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.277069 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276916    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.277069 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276929    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a564c6-7e13-4fd1-b766-4c1b477f74dc-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.277069 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276940    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a564c6-7e13-4fd1-b766-4c1b477f74dc-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.277069 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.276951    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zv5k\" (UniqueName: \"kubernetes.io/projected/73a564c6-7e13-4fd1-b766-4c1b477f74dc-kube-api-access-4zv5k\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:03:25.532919 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.532883    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"]
Sep 17 20:03:25.538475 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:25.538449    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-h5sbt"]
Sep 17 20:03:26.255363 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:26.255318    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" path="/var/lib/kubelet/pods/73a564c6-7e13-4fd1-b766-4c1b477f74dc/volumes"
Sep 17 20:03:27.730886 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:27.730848    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"]
Sep 17 20:03:30.651711 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:30.651653    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:03:40.651853 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:40.651794    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:03:42.761855 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:42.761802    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" containerName="main" containerID="cri-o://369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd" gracePeriod=45
Sep 17 20:03:50.651911 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:03:50.651864    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:04:00.652237 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:00.652188    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:04:10.652534 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:10.652483    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:04:20.652536 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:20.652477    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:04:28.408407 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.408384    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-67b9d98997-2qkj6_2068d01c-da4d-465c-ace8-840295a2ec68/main/0.log"
Sep 17 20:04:28.410073 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.410050    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:04:28.486701 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486655    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2068d01c-da4d-465c-ace8-840295a2ec68-tls-certs\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.486836 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486707    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2068d01c-da4d-465c-ace8-840295a2ec68-cabundle-cert\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.486836 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486742    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-tmp-dir\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.486836 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-home\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.486836 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-model-cache\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.486836 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486802    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-kserve-provision-location\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.486836 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxxxm\" (UniqueName: \"kubernetes.io/projected/2068d01c-da4d-465c-ace8-840295a2ec68-kube-api-access-nxxxm\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.487187 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.486861    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-dshm\") pod \"2068d01c-da4d-465c-ace8-840295a2ec68\" (UID: \"2068d01c-da4d-465c-ace8-840295a2ec68\") "
Sep 17 20:04:28.487187 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.487097    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-model-cache" (OuterVolumeSpecName: "model-cache") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:04:28.487292 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.487185    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:04:28.487348 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.487282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-home" (OuterVolumeSpecName: "home") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:04:28.487348 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.487319    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2068d01c-da4d-465c-ace8-840295a2ec68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:04:28.489009 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.488981    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2068d01c-da4d-465c-ace8-840295a2ec68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:04:28.489197 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.489173    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2068d01c-da4d-465c-ace8-840295a2ec68-kube-api-access-nxxxm" (OuterVolumeSpecName: "kube-api-access-nxxxm") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "kube-api-access-nxxxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:04:28.489308 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.489293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-dshm" (OuterVolumeSpecName: "dshm") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:04:28.506142 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.506113    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2068d01c-da4d-465c-ace8-840295a2ec68" (UID: "2068d01c-da4d-465c-ace8-840295a2ec68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:04:28.587470 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587444    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.587470 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587467    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.587604 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587478    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.587604 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587486    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxxm\" (UniqueName: \"kubernetes.io/projected/2068d01c-da4d-465c-ace8-840295a2ec68-kube-api-access-nxxxm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.587604 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587494    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.587604 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587502    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2068d01c-da4d-465c-ace8-840295a2ec68-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.587604 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587510    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2068d01c-da4d-465c-ace8-840295a2ec68-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.587604 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.587532    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2068d01c-da4d-465c-ace8-840295a2ec68-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:04:28.626071 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.626045    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-67b9d98997-2qkj6_2068d01c-da4d-465c-ace8-840295a2ec68/main/0.log"
Sep 17 20:04:28.626423 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.626402    2583 generic.go:356] "Generic (PLEG): container finished" podID="2068d01c-da4d-465c-ace8-840295a2ec68" containerID="369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd" exitCode=137
Sep 17 20:04:28.626511 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.626454    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"
Sep 17 20:04:28.626511 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.626489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" event={"ID":"2068d01c-da4d-465c-ace8-840295a2ec68","Type":"ContainerDied","Data":"369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd"}
Sep 17 20:04:28.626616 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.626540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6" event={"ID":"2068d01c-da4d-465c-ace8-840295a2ec68","Type":"ContainerDied","Data":"ac14a5cb838f591c3eb0149822f0419396cc1f768f3910a270fe8f24ce7b8dcc"}
Sep 17 20:04:28.626616 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.626558    2583 scope.go:117] "RemoveContainer" containerID="369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd"
Sep 17 20:04:28.635881 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.635862    2583 scope.go:117] "RemoveContainer" containerID="dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b"
Sep 17 20:04:28.644041 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.644025    2583 scope.go:117] "RemoveContainer" containerID="369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd"
Sep 17 20:04:28.644292 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:04:28.644273    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd\": container with ID starting with 369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd not found: ID does not exist" containerID="369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd"
Sep 17 20:04:28.644342 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.644299    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd"} err="failed to get container status \"369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd\": rpc error: code = NotFound desc = could not find container \"369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd\": container with ID starting with 369abef5d5f6c7fdca909f5572a7901b44c7113f84a53da1c811e23d952214dd not found: ID does not exist"
Sep 17 20:04:28.644342 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.644315    2583 scope.go:117] "RemoveContainer" containerID="dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b"
Sep 17 20:04:28.644531 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:04:28.644513    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b\": container with ID starting with dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b not found: ID does not exist" containerID="dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b"
Sep 17 20:04:28.644575 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.644536    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b"} err="failed to get container status \"dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b\": rpc error: code = NotFound desc = could not find container \"dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b\": container with ID starting with dfdc558005c533d56db6733d00039774d80df7188964fd2ec52a078fe331350b not found: ID does not exist"
Sep 17 20:04:28.651844 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.651824    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"]
Sep 17 20:04:28.657295 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:28.657273    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-67b9d98997-2qkj6"]
Sep 17 20:04:30.256157 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:30.256115    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" path="/var/lib/kubelet/pods/2068d01c-da4d-465c-ace8-840295a2ec68/volumes"
Sep 17 20:04:30.661115 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:30.661086    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:04:30.668559 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:30.668536    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:04:40.567067 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567035    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"]
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567470    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main"
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567481    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main"
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567498    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" containerName="storage-initializer"
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567505    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" containerName="storage-initializer"
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567512    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="storage-initializer"
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567518    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="storage-initializer"
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567534    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" containerName="main"
Sep 17 20:04:40.567593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567539    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" containerName="main"
Sep 17 20:04:40.567873 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567609    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a564c6-7e13-4fd1-b766-4c1b477f74dc" containerName="main"
Sep 17 20:04:40.567873 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.567616    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2068d01c-da4d-465c-ace8-840295a2ec68" containerName="main"
Sep 17 20:04:40.570618 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.570599    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.573317 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.573294    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 20:04:40.574443 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.574426    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 20:04:40.574555 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.574480    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 20:04:40.574555 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.574487    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-gll76\""
Sep 17 20:04:40.580779 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.580760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"]
Sep 17 20:04:40.584897 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.584879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9sng\" (UniqueName: \"kubernetes.io/projected/1447e6e6-615d-45da-af28-5939123c5f47-kube-api-access-f9sng\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.584992 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.584935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.585050 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.584985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.585050 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.585029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1447e6e6-615d-45da-af28-5939123c5f47-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.585206 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.585084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.585206 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.585147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686004 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.685977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686189 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686189 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1447e6e6-615d-45da-af28-5939123c5f47-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686189 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686189 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686189 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9sng\" (UniqueName: \"kubernetes.io/projected/1447e6e6-615d-45da-af28-5939123c5f47-kube-api-access-f9sng\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686459 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686403    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686459 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.686575 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.686510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.688472 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.688446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.688890 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.688869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1447e6e6-615d-45da-af28-5939123c5f47-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.696820 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.696797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9sng\" (UniqueName: \"kubernetes.io/projected/1447e6e6-615d-45da-af28-5939123c5f47-kube-api-access-f9sng\") pod \"rolling-upgrade-test-kserve-74b79d7f77-xc6b9\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:40.881044 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:40.880963    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:41.156848 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:41.156822    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"]
Sep 17 20:04:41.158631 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:04:41.158585    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1447e6e6_615d_45da_af28_5939123c5f47.slice/crio-2707289644ea0317cb28237c0f9f75fb13ec3cef01b4a679ac4f7be30c524a7b WatchSource:0}: Error finding container 2707289644ea0317cb28237c0f9f75fb13ec3cef01b4a679ac4f7be30c524a7b: Status 404 returned error can't find the container with id 2707289644ea0317cb28237c0f9f75fb13ec3cef01b4a679ac4f7be30c524a7b
Sep 17 20:04:41.708822 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:41.708789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" event={"ID":"1447e6e6-615d-45da-af28-5939123c5f47","Type":"ContainerStarted","Data":"f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2"}
Sep 17 20:04:41.709182 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:41.708843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" event={"ID":"1447e6e6-615d-45da-af28-5939123c5f47","Type":"ContainerStarted","Data":"2707289644ea0317cb28237c0f9f75fb13ec3cef01b4a679ac4f7be30c524a7b"}
Sep 17 20:04:41.726925 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:41.726885    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" podStartSLOduration=1.726871047 podStartE2EDuration="1.726871047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:04:41.725476829 +0000 UTC m=+3483.773356696" watchObservedRunningTime="2026-09-17 20:04:41.726871047 +0000 UTC m=+3483.774750936"
Sep 17 20:04:41.882020 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:41.881996    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:50.881951 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:50.881870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:50.894447 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:50.894418    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:04:51.776008 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:04:51.775975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:05:06.382621 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:06.382583    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 20:05:21.407523 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:21.407477    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" containerID="cri-o://1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc" gracePeriod=45
Sep 17 20:05:21.662460 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:21.662357    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": dial tcp 10.134.0.84:8000: connect: connection refused"
Sep 17 20:05:22.156711 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.156663    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:05:22.330189 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330161    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tmp-dir\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330353 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-dshm\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330353 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml5bp\" (UniqueName: \"kubernetes.io/projected/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kube-api-access-ml5bp\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330353 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330262    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kserve-provision-location\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330537 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330401    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-model-cache\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330537 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tls-certs\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330642 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330541    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-cabundle-cert\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330642 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330564    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-home\") pod \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\" (UID: \"cd73bff5-3fcc-446f-bb58-7e9052c0f24d\") "
Sep 17 20:05:22.330936 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.330905    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:22.331983 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.331209    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.331983 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.331442    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:05:22.332198 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.332176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-home" (OuterVolumeSpecName: "home") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:22.333191 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.333171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-dshm" (OuterVolumeSpecName: "dshm") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:22.333434 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.333416    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:05:22.333518 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.333444    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kube-api-access-ml5bp" (OuterVolumeSpecName: "kube-api-access-ml5bp") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "kube-api-access-ml5bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:05:22.344499 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.344476    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:22.353228 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.353206    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd73bff5-3fcc-446f-bb58-7e9052c0f24d" (UID: "cd73bff5-3fcc-446f-bb58-7e9052c0f24d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:22.432321 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.432299    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.432321 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.432320    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-cabundle-cert\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.432639 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.432329    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.432639 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.432336    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.432639 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.432343    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.432639 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.432351    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml5bp\" (UniqueName: \"kubernetes.io/projected/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kube-api-access-ml5bp\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.432639 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.432359    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd73bff5-3fcc-446f-bb58-7e9052c0f24d-kserve-provision-location\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:22.942013 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.941979    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerID="1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc" exitCode=0
Sep 17 20:05:22.942193 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.942025    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 20:05:22.942193 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.942043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cd73bff5-3fcc-446f-bb58-7e9052c0f24d","Type":"ContainerDied","Data":"1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc"}
Sep 17 20:05:22.942193 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.942110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cd73bff5-3fcc-446f-bb58-7e9052c0f24d","Type":"ContainerDied","Data":"ffd36f1796451f34581f3ecd4f806ff6de89ceb720173581fd7e980d9c088e90"}
Sep 17 20:05:22.942193 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.942137    2583 scope.go:117] "RemoveContainer" containerID="1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc"
Sep 17 20:05:22.951874 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.951858    2583 scope.go:117] "RemoveContainer" containerID="66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1"
Sep 17 20:05:22.959931 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.959916    2583 scope.go:117] "RemoveContainer" containerID="1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc"
Sep 17 20:05:22.960192 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:05:22.960176    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc\": container with ID starting with 1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc not found: ID does not exist" containerID="1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc"
Sep 17 20:05:22.960231 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.960200    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc"} err="failed to get container status \"1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc\": rpc error: code = NotFound desc = could not find container \"1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc\": container with ID starting with 1d40876b5f002d4dd749c1f7d4102c080f2a66771a38303a32710e9f4a68ebcc not found: ID does not exist"
Sep 17 20:05:22.960231 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.960216    2583 scope.go:117] "RemoveContainer" containerID="66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1"
Sep 17 20:05:22.960460 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:05:22.960442    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1\": container with ID starting with 66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1 not found: ID does not exist" containerID="66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1"
Sep 17 20:05:22.960500 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.960467    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1"} err="failed to get container status \"66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1\": rpc error: code = NotFound desc = could not find container \"66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1\": container with ID starting with 66a61fecbc71e160c9f01e7aa4a97947dd9e0697c8156f073c39a396869112c1 not found: ID does not exist"
Sep 17 20:05:22.965376 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.965354    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 20:05:22.969016 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:22.968997    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 20:05:24.254850 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:24.254814    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" path="/var/lib/kubelet/pods/cd73bff5-3fcc-446f-bb58-7e9052c0f24d/volumes"
Sep 17 20:05:25.907453 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:25.907418    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"]
Sep 17 20:05:25.917382 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:05:25.917346    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 20:05:25.917382 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:05:25.917382 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:05:25.917382 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2" cmd=["/bin/sleep","15"]
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:25.917394    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" message="" err=<
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:05:25.917407    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" podUID="1447e6e6-615d-45da-af28-5939123c5f47" containerName="main" containerID="cri-o://f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2"
Sep 17 20:05:25.917622 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:25.917451    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" podUID="1447e6e6-615d-45da-af28-5939123c5f47" containerName="main" containerID="cri-o://f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2" gracePeriod=60
Sep 17 20:05:26.260220 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.260196    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:05:26.365426 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-tmp-dir\") pod \"1447e6e6-615d-45da-af28-5939123c5f47\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") "
Sep 17 20:05:26.365570 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365445    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1447e6e6-615d-45da-af28-5939123c5f47-tls-certs\") pod \"1447e6e6-615d-45da-af28-5939123c5f47\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") "
Sep 17 20:05:26.365570 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365486    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-model-cache\") pod \"1447e6e6-615d-45da-af28-5939123c5f47\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") "
Sep 17 20:05:26.365570 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365502    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-dshm\") pod \"1447e6e6-615d-45da-af28-5939123c5f47\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") "
Sep 17 20:05:26.365570 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365555    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-home\") pod \"1447e6e6-615d-45da-af28-5939123c5f47\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") "
Sep 17 20:05:26.365835 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9sng\" (UniqueName: \"kubernetes.io/projected/1447e6e6-615d-45da-af28-5939123c5f47-kube-api-access-f9sng\") pod \"1447e6e6-615d-45da-af28-5939123c5f47\" (UID: \"1447e6e6-615d-45da-af28-5939123c5f47\") "
Sep 17 20:05:26.365835 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365658    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1447e6e6-615d-45da-af28-5939123c5f47" (UID: "1447e6e6-615d-45da-af28-5939123c5f47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:26.365835 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365822    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-model-cache" (OuterVolumeSpecName: "model-cache") pod "1447e6e6-615d-45da-af28-5939123c5f47" (UID: "1447e6e6-615d-45da-af28-5939123c5f47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:26.366011 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365836    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-home" (OuterVolumeSpecName: "home") pod "1447e6e6-615d-45da-af28-5939123c5f47" (UID: "1447e6e6-615d-45da-af28-5939123c5f47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:26.366011 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365978    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:26.366011 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.365996    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:26.366011 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.366011    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:26.367969 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.367945    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-dshm" (OuterVolumeSpecName: "dshm") pod "1447e6e6-615d-45da-af28-5939123c5f47" (UID: "1447e6e6-615d-45da-af28-5939123c5f47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:05:26.368064 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.367992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1447e6e6-615d-45da-af28-5939123c5f47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1447e6e6-615d-45da-af28-5939123c5f47" (UID: "1447e6e6-615d-45da-af28-5939123c5f47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:05:26.368132 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.368116    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1447e6e6-615d-45da-af28-5939123c5f47-kube-api-access-f9sng" (OuterVolumeSpecName: "kube-api-access-f9sng") pod "1447e6e6-615d-45da-af28-5939123c5f47" (UID: "1447e6e6-615d-45da-af28-5939123c5f47"). InnerVolumeSpecName "kube-api-access-f9sng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:05:26.466452 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.466377    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9sng\" (UniqueName: \"kubernetes.io/projected/1447e6e6-615d-45da-af28-5939123c5f47-kube-api-access-f9sng\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:26.466452 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.466416    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1447e6e6-615d-45da-af28-5939123c5f47-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:26.466452 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.466427    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1447e6e6-615d-45da-af28-5939123c5f47-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:05:26.969093 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.969060    2583 generic.go:356] "Generic (PLEG): container finished" podID="1447e6e6-615d-45da-af28-5939123c5f47" containerID="f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2" exitCode=0
Sep 17 20:05:26.969566 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.969106    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"
Sep 17 20:05:26.969566 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.969139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" event={"ID":"1447e6e6-615d-45da-af28-5939123c5f47","Type":"ContainerDied","Data":"f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2"}
Sep 17 20:05:26.969566 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.969199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9" event={"ID":"1447e6e6-615d-45da-af28-5939123c5f47","Type":"ContainerDied","Data":"2707289644ea0317cb28237c0f9f75fb13ec3cef01b4a679ac4f7be30c524a7b"}
Sep 17 20:05:26.969566 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.969230    2583 scope.go:117] "RemoveContainer" containerID="f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2"
Sep 17 20:05:26.979481 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.979466    2583 scope.go:117] "RemoveContainer" containerID="f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2"
Sep 17 20:05:26.979750 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:05:26.979733    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2\": container with ID starting with f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2 not found: ID does not exist" containerID="f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2"
Sep 17 20:05:26.979830 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.979756    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2"} err="failed to get container status \"f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2\": rpc error: code = NotFound desc = could not find container \"f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2\": container with ID starting with f9654c87e996a4abd31fd5f281342cce5445221534f2acdd06fa3e8b4417d9e2 not found: ID does not exist"
Sep 17 20:05:26.991805 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.991782    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"]
Sep 17 20:05:26.997201 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:26.997180    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-xc6b9"]
Sep 17 20:05:28.254928 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:28.254895    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1447e6e6-615d-45da-af28-5939123c5f47" path="/var/lib/kubelet/pods/1447e6e6-615d-45da-af28-5939123c5f47/volumes"
Sep 17 20:05:58.858589 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.858555    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"]
Sep 17 20:05:58.859071 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.858958    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1447e6e6-615d-45da-af28-5939123c5f47" containerName="main"
Sep 17 20:05:58.859071 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.858973    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1447e6e6-615d-45da-af28-5939123c5f47" containerName="main"
Sep 17 20:05:58.859071 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.858985    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main"
Sep 17 20:05:58.859071 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.858991    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main"
Sep 17 20:05:58.859071 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.859001    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="storage-initializer"
Sep 17 20:05:58.859071 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.859007    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="storage-initializer"
Sep 17 20:05:58.859272 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.859080    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1447e6e6-615d-45da-af28-5939123c5f47" containerName="main"
Sep 17 20:05:58.859272 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.859091    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd73bff5-3fcc-446f-bb58-7e9052c0f24d" containerName="main"
Sep 17 20:05:58.867348 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.865421    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:58.869185 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.869129    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 20:05:58.869449 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.869426    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 20:05:58.872430 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.872405    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 20:05:58.873278 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.873249    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-8x6vg\""
Sep 17 20:05:58.873545 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.873523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"]
Sep 17 20:05:58.903640 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.903607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:58.903785 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.903675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:58.903785 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.903723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:58.903785 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.903781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfw4g\" (UniqueName: \"kubernetes.io/projected/f8d0941b-69f5-473a-aad5-2c3347e7068c-kube-api-access-vfw4g\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:58.903933 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.903808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d0941b-69f5-473a-aad5-2c3347e7068c-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:58.903933 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:58.903839    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.004783 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.004759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.004885 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.004790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.004885 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.004831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfw4g\" (UniqueName: \"kubernetes.io/projected/f8d0941b-69f5-473a-aad5-2c3347e7068c-kube-api-access-vfw4g\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.004885 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.004848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d0941b-69f5-473a-aad5-2c3347e7068c-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.004885 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.004873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.005126 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.004906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.005228 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.005208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.005286 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.005223    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.005365 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.005339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.007415 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.007396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.007585 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.007567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d0941b-69f5-473a-aad5-2c3347e7068c-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.013385 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.013364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfw4g\" (UniqueName: \"kubernetes.io/projected/f8d0941b-69f5-473a-aad5-2c3347e7068c-kube-api-access-vfw4g\") pod \"scheduler-inline-config-test-kserve-8655bc757-4tfks\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.181248 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.181174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:05:59.455953 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:05:59.455925    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"]
Sep 17 20:05:59.457619 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:05:59.457595    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d0941b_69f5_473a_aad5_2c3347e7068c.slice/crio-7a3bd194fa3f647074b66e53b8d8ad030e07785075cef78886ac11ca3be4a798 WatchSource:0}: Error finding container 7a3bd194fa3f647074b66e53b8d8ad030e07785075cef78886ac11ca3be4a798: Status 404 returned error can't find the container with id 7a3bd194fa3f647074b66e53b8d8ad030e07785075cef78886ac11ca3be4a798
Sep 17 20:06:00.162124 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:00.162090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" event={"ID":"f8d0941b-69f5-473a-aad5-2c3347e7068c","Type":"ContainerStarted","Data":"837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f"}
Sep 17 20:06:00.162500 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:00.162143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" event={"ID":"f8d0941b-69f5-473a-aad5-2c3347e7068c","Type":"ContainerStarted","Data":"7a3bd194fa3f647074b66e53b8d8ad030e07785075cef78886ac11ca3be4a798"}
Sep 17 20:06:00.178867 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:00.178814    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" podStartSLOduration=2.1787973 podStartE2EDuration="2.1787973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:06:00.178608391 +0000 UTC m=+3562.226488249" watchObservedRunningTime="2026-09-17 20:06:00.1787973 +0000 UTC m=+3562.226677156"
Sep 17 20:06:00.181710 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:00.181671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:06:09.182341 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:09.182301    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:06:09.195116 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:09.195076    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:06:09.226436 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:09.226405    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:06:32.781589 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:32.781553    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"]
Sep 17 20:06:32.791284 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:06:32.791242    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 20:06:32.791284 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:06:32.791284 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:06:32.791284 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f" cmd=["/bin/sleep","15"]
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:32.791307    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" message="" err=<
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:06:32.791324    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" podUID="f8d0941b-69f5-473a-aad5-2c3347e7068c" containerName="main" containerID="cri-o://837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f"
Sep 17 20:06:32.791595 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:32.791369    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" podUID="f8d0941b-69f5-473a-aad5-2c3347e7068c" containerName="main" containerID="cri-o://837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f" gracePeriod=60
Sep 17 20:06:33.143242 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.143214    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:06:33.306244 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306212    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-model-cache\") pod \"f8d0941b-69f5-473a-aad5-2c3347e7068c\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") "
Sep 17 20:06:33.306430 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306255    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfw4g\" (UniqueName: \"kubernetes.io/projected/f8d0941b-69f5-473a-aad5-2c3347e7068c-kube-api-access-vfw4g\") pod \"f8d0941b-69f5-473a-aad5-2c3347e7068c\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") "
Sep 17 20:06:33.306430 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306300    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-tmp-dir\") pod \"f8d0941b-69f5-473a-aad5-2c3347e7068c\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") "
Sep 17 20:06:33.306430 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306368    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d0941b-69f5-473a-aad5-2c3347e7068c-tls-certs\") pod \"f8d0941b-69f5-473a-aad5-2c3347e7068c\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") "
Sep 17 20:06:33.306430 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306399    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-home\") pod \"f8d0941b-69f5-473a-aad5-2c3347e7068c\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") "
Sep 17 20:06:33.306667 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306458    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-dshm\") pod \"f8d0941b-69f5-473a-aad5-2c3347e7068c\" (UID: \"f8d0941b-69f5-473a-aad5-2c3347e7068c\") "
Sep 17 20:06:33.306667 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306549    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8d0941b-69f5-473a-aad5-2c3347e7068c" (UID: "f8d0941b-69f5-473a-aad5-2c3347e7068c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:06:33.306667 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306560    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8d0941b-69f5-473a-aad5-2c3347e7068c" (UID: "f8d0941b-69f5-473a-aad5-2c3347e7068c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:06:33.306871 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306729    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-home" (OuterVolumeSpecName: "home") pod "f8d0941b-69f5-473a-aad5-2c3347e7068c" (UID: "f8d0941b-69f5-473a-aad5-2c3347e7068c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:06:33.306925 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306897    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:06:33.306925 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306914    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:06:33.307033 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.306927    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:06:33.309140 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.309109    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d0941b-69f5-473a-aad5-2c3347e7068c-kube-api-access-vfw4g" (OuterVolumeSpecName: "kube-api-access-vfw4g") pod "f8d0941b-69f5-473a-aad5-2c3347e7068c" (UID: "f8d0941b-69f5-473a-aad5-2c3347e7068c"). InnerVolumeSpecName "kube-api-access-vfw4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:06:33.309527 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.309504    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-dshm" (OuterVolumeSpecName: "dshm") pod "f8d0941b-69f5-473a-aad5-2c3347e7068c" (UID: "f8d0941b-69f5-473a-aad5-2c3347e7068c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:06:33.309527 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.309506    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d0941b-69f5-473a-aad5-2c3347e7068c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8d0941b-69f5-473a-aad5-2c3347e7068c" (UID: "f8d0941b-69f5-473a-aad5-2c3347e7068c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:06:33.346610 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.346583    2583 generic.go:356] "Generic (PLEG): container finished" podID="f8d0941b-69f5-473a-aad5-2c3347e7068c" containerID="837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f" exitCode=0
Sep 17 20:06:33.346751 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.346625    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"
Sep 17 20:06:33.346751 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.346662    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" event={"ID":"f8d0941b-69f5-473a-aad5-2c3347e7068c","Type":"ContainerDied","Data":"837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f"}
Sep 17 20:06:33.346751 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.346745    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks" event={"ID":"f8d0941b-69f5-473a-aad5-2c3347e7068c","Type":"ContainerDied","Data":"7a3bd194fa3f647074b66e53b8d8ad030e07785075cef78886ac11ca3be4a798"}
Sep 17 20:06:33.346928 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.346776    2583 scope.go:117] "RemoveContainer" containerID="837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f"
Sep 17 20:06:33.356554 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.356534    2583 scope.go:117] "RemoveContainer" containerID="837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f"
Sep 17 20:06:33.356875 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:06:33.356854    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f\": container with ID starting with 837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f not found: ID does not exist" containerID="837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f"
Sep 17 20:06:33.356939 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.356886    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f"} err="failed to get container status \"837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f\": rpc error: code = NotFound desc = could not find container \"837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f\": container with ID starting with 837dbe72e0169716930a009fe7b9f595774875094bf2fbe6ee39eb6ec3ee4f4f not found: ID does not exist"
Sep 17 20:06:33.368388 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.368357    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"]
Sep 17 20:06:33.371767 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.371747    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4tfks"]
Sep 17 20:06:33.407567 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.407548    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8d0941b-69f5-473a-aad5-2c3347e7068c-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:06:33.407567 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.407567    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8d0941b-69f5-473a-aad5-2c3347e7068c-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:06:33.407707 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:33.407576    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfw4g\" (UniqueName: \"kubernetes.io/projected/f8d0941b-69f5-473a-aad5-2c3347e7068c-kube-api-access-vfw4g\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:06:34.257638 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:34.257592    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d0941b-69f5-473a-aad5-2c3347e7068c" path="/var/lib/kubelet/pods/f8d0941b-69f5-473a-aad5-2c3347e7068c/volumes"
Sep 17 20:06:39.359344 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.359303    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"]
Sep 17 20:06:39.360001 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.359962    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d0941b-69f5-473a-aad5-2c3347e7068c" containerName="main"
Sep 17 20:06:39.360001 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.359984    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d0941b-69f5-473a-aad5-2c3347e7068c" containerName="main"
Sep 17 20:06:39.360121 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.360098    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d0941b-69f5-473a-aad5-2c3347e7068c" containerName="main"
Sep 17 20:06:39.364584 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.364562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.368359 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.368335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-s6zwk\""
Sep 17 20:06:39.368476 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.368368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 20:06:39.368476 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.368429    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 20:06:39.368476 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.368335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 20:06:39.372165 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.372143    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"]
Sep 17 20:06:39.557157 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.557131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.557315 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.557175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10614028-e0f1-471a-965c-98bdb75239a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.557315 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.557209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.557315 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.557229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.557315 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.557260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.557315 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.557302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nzp4\" (UniqueName: \"kubernetes.io/projected/10614028-e0f1-471a-965c-98bdb75239a6-kube-api-access-5nzp4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.658640 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.658565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10614028-e0f1-471a-965c-98bdb75239a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.658640 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.658628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.658871 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.658657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.658871 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.658722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.658871 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.658745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nzp4\" (UniqueName: \"kubernetes.io/projected/10614028-e0f1-471a-965c-98bdb75239a6-kube-api-access-5nzp4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.658871 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.658783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.659207 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.659178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.659336 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.659234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.659336 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.659289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.661197 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.661182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.661483 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.661462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10614028-e0f1-471a-965c-98bdb75239a6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.667167 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.667141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nzp4\" (UniqueName: \"kubernetes.io/projected/10614028-e0f1-471a-965c-98bdb75239a6-kube-api-access-5nzp4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.675992 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.675973    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:39.950876 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:39.950851    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"]
Sep 17 20:06:39.952768 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:06:39.952736    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10614028_e0f1_471a_965c_98bdb75239a6.slice/crio-7b33a7cbf514a8d7e09708ee339c6947b4cb2f52ca2caaa36d27f7856520d9d9 WatchSource:0}: Error finding container 7b33a7cbf514a8d7e09708ee339c6947b4cb2f52ca2caaa36d27f7856520d9d9: Status 404 returned error can't find the container with id 7b33a7cbf514a8d7e09708ee339c6947b4cb2f52ca2caaa36d27f7856520d9d9
Sep 17 20:06:40.386561 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:40.386518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" event={"ID":"10614028-e0f1-471a-965c-98bdb75239a6","Type":"ContainerStarted","Data":"88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62"}
Sep 17 20:06:40.386954 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:40.386581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" event={"ID":"10614028-e0f1-471a-965c-98bdb75239a6","Type":"ContainerStarted","Data":"7b33a7cbf514a8d7e09708ee339c6947b4cb2f52ca2caaa36d27f7856520d9d9"}
Sep 17 20:06:40.415522 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:40.415474    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" podStartSLOduration=1.415460637 podStartE2EDuration="1.415460637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:06:40.414932111 +0000 UTC m=+3602.462811970" watchObservedRunningTime="2026-09-17 20:06:40.415460637 +0000 UTC m=+3602.463340517"
Sep 17 20:06:40.676953 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:40.676851    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:49.676787 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:49.676737    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:49.689320 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:49.689289    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:06:50.454005 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:06:50.453963    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:07:13.751310 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:13.751272    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"]
Sep 17 20:07:13.761168 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:07:13.761121    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 20:07:13.761168 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:07:13.761168 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:07:13.761168 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62" cmd=["/bin/sleep","15"]
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:13.761196    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" message="" err=<
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:07:13.761214    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" podUID="10614028-e0f1-471a-965c-98bdb75239a6" containerName="main" containerID="cri-o://88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62"
Sep 17 20:07:13.761431 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:13.761270    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" podUID="10614028-e0f1-471a-965c-98bdb75239a6" containerName="main" containerID="cri-o://88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62" gracePeriod=60
Sep 17 20:07:14.115733 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.115681    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:07:14.218927 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.218896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-tmp-dir\") pod \"10614028-e0f1-471a-965c-98bdb75239a6\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") "
Sep 17 20:07:14.219078 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.218935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-home\") pod \"10614028-e0f1-471a-965c-98bdb75239a6\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") "
Sep 17 20:07:14.219078 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.218981    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-dshm\") pod \"10614028-e0f1-471a-965c-98bdb75239a6\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") "
Sep 17 20:07:14.219078 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.219032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10614028-e0f1-471a-965c-98bdb75239a6-tls-certs\") pod \"10614028-e0f1-471a-965c-98bdb75239a6\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") "
Sep 17 20:07:14.219078 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.219070    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-model-cache\") pod \"10614028-e0f1-471a-965c-98bdb75239a6\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") "
Sep 17 20:07:14.219301 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.219096    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nzp4\" (UniqueName: \"kubernetes.io/projected/10614028-e0f1-471a-965c-98bdb75239a6-kube-api-access-5nzp4\") pod \"10614028-e0f1-471a-965c-98bdb75239a6\" (UID: \"10614028-e0f1-471a-965c-98bdb75239a6\") "
Sep 17 20:07:14.219301 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.219184    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10614028-e0f1-471a-965c-98bdb75239a6" (UID: "10614028-e0f1-471a-965c-98bdb75239a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:07:14.219301 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.219285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-home" (OuterVolumeSpecName: "home") pod "10614028-e0f1-471a-965c-98bdb75239a6" (UID: "10614028-e0f1-471a-965c-98bdb75239a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:07:14.219460 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.219363    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:07:14.219513 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.219458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "10614028-e0f1-471a-965c-98bdb75239a6" (UID: "10614028-e0f1-471a-965c-98bdb75239a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:07:14.221517 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.221481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-dshm" (OuterVolumeSpecName: "dshm") pod "10614028-e0f1-471a-965c-98bdb75239a6" (UID: "10614028-e0f1-471a-965c-98bdb75239a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:07:14.221517 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.221482    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10614028-e0f1-471a-965c-98bdb75239a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10614028-e0f1-471a-965c-98bdb75239a6" (UID: "10614028-e0f1-471a-965c-98bdb75239a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:07:14.221703 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.221610    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10614028-e0f1-471a-965c-98bdb75239a6-kube-api-access-5nzp4" (OuterVolumeSpecName: "kube-api-access-5nzp4") pod "10614028-e0f1-471a-965c-98bdb75239a6" (UID: "10614028-e0f1-471a-965c-98bdb75239a6"). InnerVolumeSpecName "kube-api-access-5nzp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:07:14.320828 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.320789    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:07:14.320828 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.320825    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nzp4\" (UniqueName: \"kubernetes.io/projected/10614028-e0f1-471a-965c-98bdb75239a6-kube-api-access-5nzp4\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:07:14.321035 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.320968    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:07:14.321117 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.321099    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10614028-e0f1-471a-965c-98bdb75239a6-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:07:14.321151 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.321123    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10614028-e0f1-471a-965c-98bdb75239a6-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:07:14.579146 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.579055    2583 generic.go:356] "Generic (PLEG): container finished" podID="10614028-e0f1-471a-965c-98bdb75239a6" containerID="88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62" exitCode=0
Sep 17 20:07:14.579146 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.579100    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"
Sep 17 20:07:14.579340 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.579140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" event={"ID":"10614028-e0f1-471a-965c-98bdb75239a6","Type":"ContainerDied","Data":"88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62"}
Sep 17 20:07:14.579340 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.579196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q" event={"ID":"10614028-e0f1-471a-965c-98bdb75239a6","Type":"ContainerDied","Data":"7b33a7cbf514a8d7e09708ee339c6947b4cb2f52ca2caaa36d27f7856520d9d9"}
Sep 17 20:07:14.579340 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.579218    2583 scope.go:117] "RemoveContainer" containerID="88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62"
Sep 17 20:07:14.588418 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.588401    2583 scope.go:117] "RemoveContainer" containerID="88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62"
Sep 17 20:07:14.588729 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:07:14.588711    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62\": container with ID starting with 88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62 not found: ID does not exist" containerID="88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62"
Sep 17 20:07:14.588800 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.588739    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62"} err="failed to get container status \"88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62\": rpc error: code = NotFound desc = could not find container \"88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62\": container with ID starting with 88d33542d4b4460a123fd2c2cac0e0c0577ed94152874aa7cafb2857e5736a62 not found: ID does not exist"
Sep 17 20:07:14.598160 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.598134    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"]
Sep 17 20:07:14.601457 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:14.601436    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fhvt2q"]
Sep 17 20:07:16.255188 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:16.255155    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10614028-e0f1-471a-965c-98bdb75239a6" path="/var/lib/kubelet/pods/10614028-e0f1-471a-965c-98bdb75239a6/volumes"
Sep 17 20:07:48.461579 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.461546    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"]
Sep 17 20:07:48.462002 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.461929    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10614028-e0f1-471a-965c-98bdb75239a6" containerName="main"
Sep 17 20:07:48.462002 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.461940    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="10614028-e0f1-471a-965c-98bdb75239a6" containerName="main"
Sep 17 20:07:48.462086 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.462016    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="10614028-e0f1-471a-965c-98bdb75239a6" containerName="main"
Sep 17 20:07:48.465925 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.465904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.469449 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.469423    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-cbb96\""
Sep 17 20:07:48.469583 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.469456    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 20:07:48.469583 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.469458    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 20:07:48.469583 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.469500    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 20:07:48.477636 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.477614    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"]
Sep 17 20:07:48.512632 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.512608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bgj\" (UniqueName: \"kubernetes.io/projected/b82da103-93eb-478b-b66f-b615fc76f280-kube-api-access-98bgj\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.512769 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.512645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.512769 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.512740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.512769 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.512762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.512884 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.512784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.512884 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.512799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82da103-93eb-478b-b66f-b615fc76f280-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.613536 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.613513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bgj\" (UniqueName: \"kubernetes.io/projected/b82da103-93eb-478b-b66f-b615fc76f280-kube-api-access-98bgj\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.613658 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.613551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.613658 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.613590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.613658 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.613608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.613658 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.613630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.613658 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.613650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82da103-93eb-478b-b66f-b615fc76f280-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.614067 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.614034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.614135 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.614064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.614180 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.614135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.616177 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.616154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.616472 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.616454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82da103-93eb-478b-b66f-b615fc76f280-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.622303 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.622280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bgj\" (UniqueName: \"kubernetes.io/projected/b82da103-93eb-478b-b66f-b615fc76f280-kube-api-access-98bgj\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:48.775881 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:48.775786    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:49.057873 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:49.057844    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"]
Sep 17 20:07:49.059256 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:07:49.059226    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82da103_93eb_478b_b66f_b615fc76f280.slice/crio-e11bfb48afc96bf957e34f241bb43a05ed5e2f6031b051c6bb96d05df948110c WatchSource:0}: Error finding container e11bfb48afc96bf957e34f241bb43a05ed5e2f6031b051c6bb96d05df948110c: Status 404 returned error can't find the container with id e11bfb48afc96bf957e34f241bb43a05ed5e2f6031b051c6bb96d05df948110c
Sep 17 20:07:49.769285 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:49.769246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" event={"ID":"b82da103-93eb-478b-b66f-b615fc76f280","Type":"ContainerStarted","Data":"93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d"}
Sep 17 20:07:49.769722 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:49.769304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" event={"ID":"b82da103-93eb-478b-b66f-b615fc76f280","Type":"ContainerStarted","Data":"e11bfb48afc96bf957e34f241bb43a05ed5e2f6031b051c6bb96d05df948110c"}
Sep 17 20:07:49.776548 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:49.776521    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:49.790865 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:49.790812    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" podStartSLOduration=1.790796904 podStartE2EDuration="1.790796904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:07:49.788619235 +0000 UTC m=+3671.836499092" watchObservedRunningTime="2026-09-17 20:07:49.790796904 +0000 UTC m=+3671.838676778"
Sep 17 20:07:58.776657 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:58.776617    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:58.789502 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:58.789471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:07:58.831404 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:07:58.831375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:08:22.336809 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.336768    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"]
Sep 17 20:08:22.346890 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:08:22.346853    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 20:08:22.346890 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:08:22.346890 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:08:22.346890 ip-10-0-128-131 kubenswrapper[2583]:  > containerID="93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d" cmd=["/bin/sleep","15"]
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.346908    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" message="" err=<
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]:  >
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:08:22.346924    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" podUID="b82da103-93eb-478b-b66f-b615fc76f280" containerName="main" containerID="cri-o://93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d"
Sep 17 20:08:22.347129 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.346964    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" podUID="b82da103-93eb-478b-b66f-b615fc76f280" containerName="main" containerID="cri-o://93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d" gracePeriod=60
Sep 17 20:08:22.704316 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.704291    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:08:22.786914 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.786886    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-dshm\") pod \"b82da103-93eb-478b-b66f-b615fc76f280\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") "
Sep 17 20:08:22.787060 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.786920    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82da103-93eb-478b-b66f-b615fc76f280-tls-certs\") pod \"b82da103-93eb-478b-b66f-b615fc76f280\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") "
Sep 17 20:08:22.787060 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.786952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-model-cache\") pod \"b82da103-93eb-478b-b66f-b615fc76f280\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") "
Sep 17 20:08:22.787060 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.786995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-home\") pod \"b82da103-93eb-478b-b66f-b615fc76f280\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") "
Sep 17 20:08:22.787060 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.787011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-tmp-dir\") pod \"b82da103-93eb-478b-b66f-b615fc76f280\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") "
Sep 17 20:08:22.787254 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.787068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98bgj\" (UniqueName: \"kubernetes.io/projected/b82da103-93eb-478b-b66f-b615fc76f280-kube-api-access-98bgj\") pod \"b82da103-93eb-478b-b66f-b615fc76f280\" (UID: \"b82da103-93eb-478b-b66f-b615fc76f280\") "
Sep 17 20:08:22.787330 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.787298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-model-cache" (OuterVolumeSpecName: "model-cache") pod "b82da103-93eb-478b-b66f-b615fc76f280" (UID: "b82da103-93eb-478b-b66f-b615fc76f280"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:08:22.787330 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.787313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b82da103-93eb-478b-b66f-b615fc76f280" (UID: "b82da103-93eb-478b-b66f-b615fc76f280"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:08:22.787448 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.787322    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-home" (OuterVolumeSpecName: "home") pod "b82da103-93eb-478b-b66f-b615fc76f280" (UID: "b82da103-93eb-478b-b66f-b615fc76f280"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:08:22.789362 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.789335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82da103-93eb-478b-b66f-b615fc76f280-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b82da103-93eb-478b-b66f-b615fc76f280" (UID: "b82da103-93eb-478b-b66f-b615fc76f280"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:08:22.789476 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.789370    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-dshm" (OuterVolumeSpecName: "dshm") pod "b82da103-93eb-478b-b66f-b615fc76f280" (UID: "b82da103-93eb-478b-b66f-b615fc76f280"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:08:22.789538 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.789476    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82da103-93eb-478b-b66f-b615fc76f280-kube-api-access-98bgj" (OuterVolumeSpecName: "kube-api-access-98bgj") pod "b82da103-93eb-478b-b66f-b615fc76f280" (UID: "b82da103-93eb-478b-b66f-b615fc76f280"). InnerVolumeSpecName "kube-api-access-98bgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:08:22.887872 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.887816    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-dshm\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:08:22.887872 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.887838    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b82da103-93eb-478b-b66f-b615fc76f280-tls-certs\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:08:22.887872 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.887848    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-model-cache\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:08:22.887872 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.887856    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-home\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:08:22.888064 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.887881    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b82da103-93eb-478b-b66f-b615fc76f280-tmp-dir\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:08:22.888064 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.887890    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98bgj\" (UniqueName: \"kubernetes.io/projected/b82da103-93eb-478b-b66f-b615fc76f280-kube-api-access-98bgj\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:08:22.951431 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.951401    2583 generic.go:356] "Generic (PLEG): container finished" podID="b82da103-93eb-478b-b66f-b615fc76f280" containerID="93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d" exitCode=0
Sep 17 20:08:22.951545 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.951452    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"
Sep 17 20:08:22.951545 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.951489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" event={"ID":"b82da103-93eb-478b-b66f-b615fc76f280","Type":"ContainerDied","Data":"93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d"}
Sep 17 20:08:22.951545 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.951541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw" event={"ID":"b82da103-93eb-478b-b66f-b615fc76f280","Type":"ContainerDied","Data":"e11bfb48afc96bf957e34f241bb43a05ed5e2f6031b051c6bb96d05df948110c"}
Sep 17 20:08:22.951657 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.951564    2583 scope.go:117] "RemoveContainer" containerID="93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d"
Sep 17 20:08:22.961016 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.961002    2583 scope.go:117] "RemoveContainer" containerID="93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d"
Sep 17 20:08:22.961290 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:08:22.961256    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d\": container with ID starting with 93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d not found: ID does not exist" containerID="93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d"
Sep 17 20:08:22.961377 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.961297    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d"} err="failed to get container status \"93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d\": rpc error: code = NotFound desc = could not find container \"93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d\": container with ID starting with 93c9663e926c8ef09712084e23387fa3a6d821333ea95508c25de182d078668d not found: ID does not exist"
Sep 17 20:08:22.973497 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.973475    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"]
Sep 17 20:08:22.977558 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:22.977538    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vwbfw"]
Sep 17 20:08:24.255145 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:08:24.255106    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b82da103-93eb-478b-b66f-b615fc76f280" path="/var/lib/kubelet/pods/b82da103-93eb-478b-b66f-b615fc76f280/volumes"
Sep 17 20:09:15.737765 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.737725    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n9d82/must-gather-bnvff"]
Sep 17 20:09:15.738258 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.738134    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82da103-93eb-478b-b66f-b615fc76f280" containerName="main"
Sep 17 20:09:15.738258 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.738146    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82da103-93eb-478b-b66f-b615fc76f280" containerName="main"
Sep 17 20:09:15.738258 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.738201    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82da103-93eb-478b-b66f-b615fc76f280" containerName="main"
Sep 17 20:09:15.741222 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.741204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:15.743876 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.743844    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9d82\"/\"kube-root-ca.crt\""
Sep 17 20:09:15.745118 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.745096    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n9d82\"/\"default-dockercfg-fcdwm\""
Sep 17 20:09:15.745234 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.745129    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9d82\"/\"openshift-service-ca.crt\""
Sep 17 20:09:15.751488 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.751463    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9d82/must-gather-bnvff"]
Sep 17 20:09:15.820601 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.820561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp9w9\" (UniqueName: \"kubernetes.io/projected/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-kube-api-access-xp9w9\") pod \"must-gather-bnvff\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") " pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:15.820601 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.820608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-must-gather-output\") pod \"must-gather-bnvff\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") " pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:15.921984 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.921944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp9w9\" (UniqueName: \"kubernetes.io/projected/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-kube-api-access-xp9w9\") pod \"must-gather-bnvff\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") " pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:15.922163 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.921992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-must-gather-output\") pod \"must-gather-bnvff\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") " pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:15.922374 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.922353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-must-gather-output\") pod \"must-gather-bnvff\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") " pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:15.934859 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:15.934826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp9w9\" (UniqueName: \"kubernetes.io/projected/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-kube-api-access-xp9w9\") pod \"must-gather-bnvff\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") " pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:16.050897 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:16.050849    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:16.322598 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:16.322564    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9d82/must-gather-bnvff"]
Sep 17 20:09:16.324507 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:09:16.324475    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0bc5e55_d39f_4d42_b3d8_9ec91675772a.slice/crio-e3dca93c92ecbb61fc674c46260bd62ab92e9414cacd9e303e2cd7fbf5fced50 WatchSource:0}: Error finding container e3dca93c92ecbb61fc674c46260bd62ab92e9414cacd9e303e2cd7fbf5fced50: Status 404 returned error can't find the container with id e3dca93c92ecbb61fc674c46260bd62ab92e9414cacd9e303e2cd7fbf5fced50
Sep 17 20:09:16.326751 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:16.326734    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:09:17.270877 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:17.270794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9d82/must-gather-bnvff" event={"ID":"e0bc5e55-d39f-4d42-b3d8-9ec91675772a","Type":"ContainerStarted","Data":"e3dca93c92ecbb61fc674c46260bd62ab92e9414cacd9e303e2cd7fbf5fced50"}
Sep 17 20:09:23.314873 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:23.314822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9d82/must-gather-bnvff" event={"ID":"e0bc5e55-d39f-4d42-b3d8-9ec91675772a","Type":"ContainerStarted","Data":"b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f"}
Sep 17 20:09:23.315291 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:23.314882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9d82/must-gather-bnvff" event={"ID":"e0bc5e55-d39f-4d42-b3d8-9ec91675772a","Type":"ContainerStarted","Data":"b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336"}
Sep 17 20:09:23.329534 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:23.329465    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n9d82/must-gather-bnvff" podStartSLOduration=1.766141645 podStartE2EDuration="8.329450394s" totalImagesPullingTime="6.563308749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:09:23.329254861 +0000 UTC m=+3765.377134729" watchObservedRunningTime="2026-09-17 20:09:23.329450394 +0000 UTC m=+3765.377330322"
Sep 17 20:09:41.405734 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:41.405651    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-8pjhs_d464d6ee-96bd-4e67-9476-854efbba0abd/manager/0.log"
Sep 17 20:09:41.432924 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:41.432896    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9qrk2_859ac01d-89f9-4947-b62c-7806c5833e59/kuadrant-console-plugin/0.log"
Sep 17 20:09:41.517663 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:41.517636    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-xd5cs_24fe1c6f-7a85-4a70-b4aa-b76bab1314d2/manager/0.log"
Sep 17 20:09:41.526708 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:41.526663    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-nlwm5_0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773/limitador/0.log"
Sep 17 20:09:41.542164 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:41.542143    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-hx9zc_16ae978c-1626-473a-8b4b-aa0b04747f98/manager/0.log"
Sep 17 20:09:44.448052 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:44.448018    2583 generic.go:356] "Generic (PLEG): container finished" podID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerID="b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336" exitCode=0
Sep 17 20:09:44.448478 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:44.448104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9d82/must-gather-bnvff" event={"ID":"e0bc5e55-d39f-4d42-b3d8-9ec91675772a","Type":"ContainerDied","Data":"b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336"}
Sep 17 20:09:44.448478 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:44.448398    2583 scope.go:117] "RemoveContainer" containerID="b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336"
Sep 17 20:09:44.668114 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:44.668088    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n9d82_must-gather-bnvff_e0bc5e55-d39f-4d42-b3d8-9ec91675772a/gather/0.log"
Sep 17 20:09:45.292584 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.292512    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rhr8t/must-gather-z6ks6"]
Sep 17 20:09:45.296935 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.296916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.300162 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.300132    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rhr8t\"/\"openshift-service-ca.crt\""
Sep 17 20:09:45.300460 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.300436    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rhr8t\"/\"kube-root-ca.crt\""
Sep 17 20:09:45.301274 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.301251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rhr8t\"/\"default-dockercfg-cks9b\""
Sep 17 20:09:45.306332 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.306301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhr8t/must-gather-z6ks6"]
Sep 17 20:09:45.387627 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.387600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/17f13446-180e-4920-a607-2ab25acda6a6-must-gather-output\") pod \"must-gather-z6ks6\" (UID: \"17f13446-180e-4920-a607-2ab25acda6a6\") " pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.387745 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.387635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6xm\" (UniqueName: \"kubernetes.io/projected/17f13446-180e-4920-a607-2ab25acda6a6-kube-api-access-tg6xm\") pod \"must-gather-z6ks6\" (UID: \"17f13446-180e-4920-a607-2ab25acda6a6\") " pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.488874 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.488849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/17f13446-180e-4920-a607-2ab25acda6a6-must-gather-output\") pod \"must-gather-z6ks6\" (UID: \"17f13446-180e-4920-a607-2ab25acda6a6\") " pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.489156 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.488883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6xm\" (UniqueName: \"kubernetes.io/projected/17f13446-180e-4920-a607-2ab25acda6a6-kube-api-access-tg6xm\") pod \"must-gather-z6ks6\" (UID: \"17f13446-180e-4920-a607-2ab25acda6a6\") " pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.489198 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.489172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/17f13446-180e-4920-a607-2ab25acda6a6-must-gather-output\") pod \"must-gather-z6ks6\" (UID: \"17f13446-180e-4920-a607-2ab25acda6a6\") " pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.497431 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.497402    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6xm\" (UniqueName: \"kubernetes.io/projected/17f13446-180e-4920-a607-2ab25acda6a6-kube-api-access-tg6xm\") pod \"must-gather-z6ks6\" (UID: \"17f13446-180e-4920-a607-2ab25acda6a6\") " pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.611982 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.611960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhr8t/must-gather-z6ks6"
Sep 17 20:09:45.888701 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:45.888660    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhr8t/must-gather-z6ks6"]
Sep 17 20:09:46.461525 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:46.461487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhr8t/must-gather-z6ks6" event={"ID":"17f13446-180e-4920-a607-2ab25acda6a6","Type":"ContainerStarted","Data":"10b4ebd6e2d4a86f1cccbf63a44079407d991df7c61f04d7c35d6a9ab0779b58"}
Sep 17 20:09:47.471219 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:47.471171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhr8t/must-gather-z6ks6" event={"ID":"17f13446-180e-4920-a607-2ab25acda6a6","Type":"ContainerStarted","Data":"29803cb2625960a93c30035d768698511c961457437c94db66793c80ddcd84b5"}
Sep 17 20:09:47.471731 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:47.471236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhr8t/must-gather-z6ks6" event={"ID":"17f13446-180e-4920-a607-2ab25acda6a6","Type":"ContainerStarted","Data":"c6162b597dc7d39121386e6296b01e4686f918d4e695416b9cc78bd42574ad25"}
Sep 17 20:09:48.412260 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:48.412227    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tl6xh_0d065b95-a94c-4a9f-a0b1-5f20f660c173/global-pull-secret-syncer/0.log"
Sep 17 20:09:48.513094 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:48.513061    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4qvt9_f016d2b8-1a3f-47ee-885a-8b43bdd1383e/konnectivity-agent/0.log"
Sep 17 20:09:48.575317 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:48.575287    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-131.ec2.internal_e72bea0306cf2b83af0585e51b861b54/haproxy/0.log"
Sep 17 20:09:50.074906 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.074844    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rhr8t/must-gather-z6ks6" podStartSLOduration=4.137824524 podStartE2EDuration="5.074818357s" totalImagesPullingTime="936.993833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:09:47.490854497 +0000 UTC m=+3789.538734355" watchObservedRunningTime="2026-09-17 20:09:50.074818357 +0000 UTC m=+3792.122698215"
Sep 17 20:09:50.082237 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.082203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-n9d82/must-gather-bnvff"]
Sep 17 20:09:50.082496 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.082480    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-n9d82/must-gather-bnvff"]
Sep 17 20:09:50.083137 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.082794    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-n9d82/must-gather-bnvff" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerName="copy" containerID="cri-o://b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f" gracePeriod=2
Sep 17 20:09:50.085181 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:09:50.085152    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bnvff\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n9d82\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:50.268711 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:09:50.268354    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bnvff\" is forbidden: User \"system:node:ip-10-0-128-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-n9d82\": no relationship found between node 'ip-10-0-128-131.ec2.internal' and this object" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:50.451705 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.450807    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n9d82_must-gather-bnvff_e0bc5e55-d39f-4d42-b3d8-9ec91675772a/copy/0.log"
Sep 17 20:09:50.455881 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.454461    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:50.495461 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.494175    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-n9d82_must-gather-bnvff_e0bc5e55-d39f-4d42-b3d8-9ec91675772a/copy/0.log"
Sep 17 20:09:50.495461 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.494664    2583 generic.go:356] "Generic (PLEG): container finished" podID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerID="b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f" exitCode=143
Sep 17 20:09:50.495461 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.494741    2583 scope.go:117] "RemoveContainer" containerID="b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f"
Sep 17 20:09:50.495461 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.494909    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9d82/must-gather-bnvff"
Sep 17 20:09:50.529249 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.529214    2583 scope.go:117] "RemoveContainer" containerID="b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336"
Sep 17 20:09:50.538720 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.536388    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-must-gather-output\") pod \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") "
Sep 17 20:09:50.538720 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.536459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp9w9\" (UniqueName: \"kubernetes.io/projected/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-kube-api-access-xp9w9\") pod \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\" (UID: \"e0bc5e55-d39f-4d42-b3d8-9ec91675772a\") "
Sep 17 20:09:50.545893 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.545847    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-kube-api-access-xp9w9" (OuterVolumeSpecName: "kube-api-access-xp9w9") pod "e0bc5e55-d39f-4d42-b3d8-9ec91675772a" (UID: "e0bc5e55-d39f-4d42-b3d8-9ec91675772a"). InnerVolumeSpecName "kube-api-access-xp9w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:09:50.556440 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.556380    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e0bc5e55-d39f-4d42-b3d8-9ec91675772a" (UID: "e0bc5e55-d39f-4d42-b3d8-9ec91675772a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:09:50.591886 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.591840    2583 scope.go:117] "RemoveContainer" containerID="b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f"
Sep 17 20:09:50.592344 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:09:50.592310    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f\": container with ID starting with b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f not found: ID does not exist" containerID="b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f"
Sep 17 20:09:50.592456 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.592350    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f"} err="failed to get container status \"b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f\": rpc error: code = NotFound desc = could not find container \"b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f\": container with ID starting with b18301a663a09181c554f510dada4750b9d07a0daee92084293702bae478f19f not found: ID does not exist"
Sep 17 20:09:50.592456 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.592375    2583 scope.go:117] "RemoveContainer" containerID="b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336"
Sep 17 20:09:50.592790 ip-10-0-128-131 kubenswrapper[2583]: E0917 20:09:50.592754    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336\": container with ID starting with b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336 not found: ID does not exist" containerID="b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336"
Sep 17 20:09:50.592879 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.592786    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336"} err="failed to get container status \"b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336\": rpc error: code = NotFound desc = could not find container \"b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336\": container with ID starting with b83d8d486347fb3570b5c8bf0c33d93cf44c5a42163a39f25d6b6241440de336 not found: ID does not exist"
Sep 17 20:09:50.637924 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.637851    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-must-gather-output\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:09:50.637924 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:50.637897    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp9w9\" (UniqueName: \"kubernetes.io/projected/e0bc5e55-d39f-4d42-b3d8-9ec91675772a-kube-api-access-xp9w9\") on node \"ip-10-0-128-131.ec2.internal\" DevicePath \"\""
Sep 17 20:09:52.265393 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:52.265348    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" path="/var/lib/kubelet/pods/e0bc5e55-d39f-4d42-b3d8-9ec91675772a/volumes"
Sep 17 20:09:53.621408 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:53.621370    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-8pjhs_d464d6ee-96bd-4e67-9476-854efbba0abd/manager/0.log"
Sep 17 20:09:53.665302 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:53.665269    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9qrk2_859ac01d-89f9-4947-b62c-7806c5833e59/kuadrant-console-plugin/0.log"
Sep 17 20:09:53.792712 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:53.791822    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-xd5cs_24fe1c6f-7a85-4a70-b4aa-b76bab1314d2/manager/0.log"
Sep 17 20:09:53.824499 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:53.824468    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-nlwm5_0d377d7b-5d46-4d39-a2a6-5cc0f1aaf773/limitador/0.log"
Sep 17 20:09:53.856002 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:53.855957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-hx9zc_16ae978c-1626-473a-8b4b-aa0b04747f98/manager/0.log"
Sep 17 20:09:55.085256 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.085229    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_090fc964-f14c-4de7-9ac4-5b87652938e3/alertmanager/0.log"
Sep 17 20:09:55.114497 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.114463    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_090fc964-f14c-4de7-9ac4-5b87652938e3/config-reloader/0.log"
Sep 17 20:09:55.140385 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.140353    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_090fc964-f14c-4de7-9ac4-5b87652938e3/kube-rbac-proxy-web/0.log"
Sep 17 20:09:55.168041 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.168002    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_090fc964-f14c-4de7-9ac4-5b87652938e3/kube-rbac-proxy/0.log"
Sep 17 20:09:55.192225 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.192193    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_090fc964-f14c-4de7-9ac4-5b87652938e3/kube-rbac-proxy-metric/0.log"
Sep 17 20:09:55.216497 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.216471    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_090fc964-f14c-4de7-9ac4-5b87652938e3/prom-label-proxy/0.log"
Sep 17 20:09:55.242872 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.242842    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_090fc964-f14c-4de7-9ac4-5b87652938e3/init-config-reloader/0.log"
Sep 17 20:09:55.342828 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.342748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-jlb8k_aeeef8c0-6a47-4686-bfae-6a0b500d09e0/kube-state-metrics/0.log"
Sep 17 20:09:55.362602 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.362567    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-jlb8k_aeeef8c0-6a47-4686-bfae-6a0b500d09e0/kube-rbac-proxy-main/0.log"
Sep 17 20:09:55.390017 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.389982    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-jlb8k_aeeef8c0-6a47-4686-bfae-6a0b500d09e0/kube-rbac-proxy-self/0.log"
Sep 17 20:09:55.567539 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.567502    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pnkj5_9bf21b45-0fc6-4ff3-8a74-d5b9b0423522/node-exporter/0.log"
Sep 17 20:09:55.590480 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.590447    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pnkj5_9bf21b45-0fc6-4ff3-8a74-d5b9b0423522/kube-rbac-proxy/0.log"
Sep 17 20:09:55.614851 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:55.614763    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pnkj5_9bf21b45-0fc6-4ff3-8a74-d5b9b0423522/init-textfile/0.log"
Sep 17 20:09:56.142296 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.142241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f7957bd96-ptrpg_19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7/telemeter-client/0.log"
Sep 17 20:09:56.174040 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.174001    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f7957bd96-ptrpg_19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7/reload/0.log"
Sep 17 20:09:56.199062 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.199020    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f7957bd96-ptrpg_19b0be20-4e4f-44d1-82a7-c95f0cc1fbb7/kube-rbac-proxy/0.log"
Sep 17 20:09:56.234069 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.234008    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb8c8f5c-sbrt6_4d120838-aa40-46bb-bc83-cc659f36da0c/thanos-query/0.log"
Sep 17 20:09:56.264321 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.264237    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb8c8f5c-sbrt6_4d120838-aa40-46bb-bc83-cc659f36da0c/kube-rbac-proxy-web/0.log"
Sep 17 20:09:56.290437 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.290388    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb8c8f5c-sbrt6_4d120838-aa40-46bb-bc83-cc659f36da0c/kube-rbac-proxy/0.log"
Sep 17 20:09:56.315919 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.315829    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb8c8f5c-sbrt6_4d120838-aa40-46bb-bc83-cc659f36da0c/prom-label-proxy/0.log"
Sep 17 20:09:56.339868 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.339813    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb8c8f5c-sbrt6_4d120838-aa40-46bb-bc83-cc659f36da0c/kube-rbac-proxy-rules/0.log"
Sep 17 20:09:56.370737 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:56.370645    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb8c8f5c-sbrt6_4d120838-aa40-46bb-bc83-cc659f36da0c/kube-rbac-proxy-metrics/0.log"
Sep 17 20:09:58.545092 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.545058    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"]
Sep 17 20:09:58.546318 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.546293    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerName="gather"
Sep 17 20:09:58.546667 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.546651    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerName="gather"
Sep 17 20:09:58.546851 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.546839    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerName="copy"
Sep 17 20:09:58.546936 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.546927    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerName="copy"
Sep 17 20:09:58.547149 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.547128    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerName="gather"
Sep 17 20:09:58.547149 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.547154    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bc5e55-d39f-4d42-b3d8-9ec91675772a" containerName="copy"
Sep 17 20:09:58.553784 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.553763    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.556248 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.556224    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"]
Sep 17 20:09:58.594707 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.594657    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-645bf46b68-88fhx_7fbc1c4a-b3e2-42ca-b5aa-0d5107fbe447/console/0.log"
Sep 17 20:09:58.632142 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.632101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-proc\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.632337 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.632176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-sys\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.632337 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.632222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-podres\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.632337 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.632279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mf6\" (UniqueName: \"kubernetes.io/projected/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-kube-api-access-h4mf6\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.632337 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.632303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-lib-modules\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.733086 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.733041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-proc\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.733385 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.733360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-sys\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.733555 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.733538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-podres\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.733733 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.733712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mf6\" (UniqueName: \"kubernetes.io/projected/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-kube-api-access-h4mf6\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.733872 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.733855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-lib-modules\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.734176 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.734154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-lib-modules\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.734366 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.734340    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-proc\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.734459 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.734396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-sys\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.734517 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.734498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-podres\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.743782 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.743757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mf6\" (UniqueName: \"kubernetes.io/projected/9d5247bc-a5ca-4590-83ac-89bd84e92a7e-kube-api-access-h4mf6\") pod \"perf-node-gather-daemonset-27lg4\" (UID: \"9d5247bc-a5ca-4590-83ac-89bd84e92a7e\") " pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:58.866593 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:58.866546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:59.154336 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:59.154304    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"]
Sep 17 20:09:59.156336 ip-10-0-128-131 kubenswrapper[2583]: W0917 20:09:59.156305    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d5247bc_a5ca_4590_83ac_89bd84e92a7e.slice/crio-eb5960f32e30605653cbf38e164de7694ddb9fd933918db92cc3025fe1b20127 WatchSource:0}: Error finding container eb5960f32e30605653cbf38e164de7694ddb9fd933918db92cc3025fe1b20127: Status 404 returned error can't find the container with id eb5960f32e30605653cbf38e164de7694ddb9fd933918db92cc3025fe1b20127
Sep 17 20:09:59.581426 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:59.581377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4" event={"ID":"9d5247bc-a5ca-4590-83ac-89bd84e92a7e","Type":"ContainerStarted","Data":"6e70ee6b54fed4510e8e37e109abfc0f113bbf69be22c5fba497d3fddd60b9af"}
Sep 17 20:09:59.581917 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:59.581445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4" event={"ID":"9d5247bc-a5ca-4590-83ac-89bd84e92a7e","Type":"ContainerStarted","Data":"eb5960f32e30605653cbf38e164de7694ddb9fd933918db92cc3025fe1b20127"}
Sep 17 20:09:59.581917 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:59.581478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:09:59.601274 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:09:59.601221    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4" podStartSLOduration=1.601205301 podStartE2EDuration="1.601205301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:09:59.597721818 +0000 UTC m=+3801.645601676" watchObservedRunningTime="2026-09-17 20:09:59.601205301 +0000 UTC m=+3801.649085214"
Sep 17 20:10:00.247960 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:00.247930    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwqkr_339bcb5f-ffaa-42f3-b10f-81dc7e391a5b/dns/0.log"
Sep 17 20:10:00.270587 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:00.270558    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pwqkr_339bcb5f-ffaa-42f3-b10f-81dc7e391a5b/kube-rbac-proxy/0.log"
Sep 17 20:10:00.398276 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:00.398241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-njvgl_9e17c207-4488-49ec-b117-644ad958d38c/dns-node-resolver/0.log"
Sep 17 20:10:01.024282 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:01.024251    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77bb5566d-pvt2v_5052a006-9c61-40e2-be09-924837e29bb0/registry/0.log"
Sep 17 20:10:01.048045 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:01.048019    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrd25_6efce3ed-bfc7-4d90-bc0c-af87cba720c8/node-ca/0.log"
Sep 17 20:10:02.765423 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:02.765390    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42dpm_b307565b-358c-426e-b984-e28122a91bb1/serve-healthcheck-canary/0.log"
Sep 17 20:10:03.472812 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:03.472780    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmgtc_822619f9-b256-49c2-be08-b881baa6346e/kube-rbac-proxy/0.log"
Sep 17 20:10:03.496483 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:03.496459    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmgtc_822619f9-b256-49c2-be08-b881baa6346e/exporter/0.log"
Sep 17 20:10:03.520603 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:03.520579    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmgtc_822619f9-b256-49c2-be08-b881baa6346e/extractor/0.log"
Sep 17 20:10:05.598751 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:05.598719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rhr8t/perf-node-gather-daemonset-27lg4"
Sep 17 20:10:06.523591 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:06.523562    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gnnrq_e06d5ff7-b02f-4706-84c6-ed9229ae262a/openshift-lws-operator/0.log"
Sep 17 20:10:07.196908 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:07.196868    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-66576f5db6-6qjz6_0dcc16c3-3650-4c98-b349-c52a27e4efba/manager/0.log"
Sep 17 20:10:07.592380 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:07.592341    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-b64f7dcc6-7mn8m_094fe571-f8e5-4841-8b88-feca944b2015/manager/0.log"
Sep 17 20:10:07.616797 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:07.616765    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-m7tqm_3127ea0c-484e-4134-8c0d-cc5cefaee075/server/0.log"
Sep 17 20:10:08.174678 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:08.174645    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8p82n_202f4200-5551-44ca-9493-ca52c5fe4c51/s3-init/0.log"
Sep 17 20:10:08.207394 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:08.207328    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8p82n_202f4200-5551-44ca-9493-ca52c5fe4c51/download-hf-model/0.log"
Sep 17 20:10:08.259736 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:08.259669    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-sr7zz_4f7974ad-93f1-4883-9f1a-43c708c6b4cb/seaweedfs/0.log"
Sep 17 20:10:15.267607 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.267579    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zz8lm_8e76a145-f19d-4439-99c3-3501aae73b8d/kube-multus-additional-cni-plugins/0.log"
Sep 17 20:10:15.291319 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.291295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zz8lm_8e76a145-f19d-4439-99c3-3501aae73b8d/egress-router-binary-copy/0.log"
Sep 17 20:10:15.314802 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.314781    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zz8lm_8e76a145-f19d-4439-99c3-3501aae73b8d/cni-plugins/0.log"
Sep 17 20:10:15.337982 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.337956    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zz8lm_8e76a145-f19d-4439-99c3-3501aae73b8d/bond-cni-plugin/0.log"
Sep 17 20:10:15.362236 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.362203    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zz8lm_8e76a145-f19d-4439-99c3-3501aae73b8d/routeoverride-cni/0.log"
Sep 17 20:10:15.388132 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.388099    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zz8lm_8e76a145-f19d-4439-99c3-3501aae73b8d/whereabouts-cni-bincopy/0.log"
Sep 17 20:10:15.412154 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.412116    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zz8lm_8e76a145-f19d-4439-99c3-3501aae73b8d/whereabouts-cni/0.log"
Sep 17 20:10:15.500404 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.500377    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qjbj4_094a9e6e-f49f-4b2b-bed0-b49e090d090d/kube-multus/0.log"
Sep 17 20:10:15.613863 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.613841    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hpls8_35716ede-fc12-454a-a045-aca7fee96b1e/network-metrics-daemon/0.log"
Sep 17 20:10:15.637035 ip-10-0-128-131 kubenswrapper[2583]: I0917 20:10:15.637009    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hpls8_35716ede-fc12-454a-a045-aca7fee96b1e/kube-rbac-proxy/0.log"