Sep 18 21:50:41.636244 ip-10-0-142-209 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 21:50:44.763917 ip-10-0-142-209 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:50:44.763917 ip-10-0-142-209 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 21:50:44.763917 ip-10-0-142-209 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:50:44.763917 ip-10-0-142-209 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 21:50:44.763917 ip-10-0-142-209 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 21:50:44.764676 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.764559    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 21:50:44.767991 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.767976    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:44.767991 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.767991    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.767995    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.767998    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768001    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768005    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768008    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768011    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768014    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768016    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768019    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768022    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768024    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768029    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768033    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768036    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768039    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768042    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768044    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768052    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768055    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:44.768058 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768058    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768060    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768063    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768066    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768069    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768072    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768075    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768079    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768082    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768085    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768088    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768091    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768094    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768096    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768099    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768101    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768104    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768107    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768109    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768112    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:44.768526 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768115    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768118    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768121    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768124    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768126    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768129    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768131    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768134    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768136    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768139    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768141    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768144    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768146    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768148    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768151    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768154    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768157    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768160    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768162    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:44.769075 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768165    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768168    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768170    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768173    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768175    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768178    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768180    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768183    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768185    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768188    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768191    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768196    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768199    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768202    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768204    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768208    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768211    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768213    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768216    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:44.769521 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768218    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768221    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768224    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768226    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768229    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768231    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768234    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768237    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768239    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768242    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768245    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768248    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768250    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768253    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768255    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768258    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768260    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768263    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768265    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.768268    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:44.769979 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769597    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769604    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769608    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769610    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769613    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769617    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769620    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769622    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769625    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769627    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769630    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769633    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769635    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769638    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769640    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769643    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769646    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769648    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769651    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769653    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:44.770443 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769656    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769659    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769661    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769664    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769666    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769669    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769671    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769674    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769676    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769679    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769681    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769683    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769687    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769691    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769693    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769696    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769698    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769700    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769704    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769706    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:44.770915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769709    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769711    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769714    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769716    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769720    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769722    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769726    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769728    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769731    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769733    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769736    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769738    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769742    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769745    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769747    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769750    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769752    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769755    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769758    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769761    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:44.771439 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769763    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769766    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769769    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769771    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769775    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769778    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769781    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769784    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769786    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769790    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769793    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769796    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769798    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769801    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769803    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769805    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769808    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769811    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769813    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:44.771964 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769816    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769818    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769821    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769823    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769826    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769830    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769833    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769836    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769839    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769841    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769844    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769847    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769850    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769852    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769855    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769858    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769860    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769862    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769865    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:44.772426 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.769867    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770233    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770238    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770241    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770245    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770248    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770251    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770254    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770256    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770259    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770262    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770264    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770267    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770270    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770272    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770275    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770277    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770280    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770283    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770285    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:44.772889 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770288    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770290    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770293    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770295    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770299    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770303    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770306    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770309    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770312    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770315    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770318    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770321    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770323    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770326    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770328    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770331    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770333    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770336    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770339    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770341    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:44.773433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770344    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770346    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770348    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770351    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770354    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770356    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770360    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770363    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770366    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770369    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770371    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770374    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770376    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770379    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770381    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770384    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770386    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770389    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770393    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770396    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:44.773915 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770398    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770401    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770404    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770406    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770409    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770411    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770414    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770416    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770419    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770421    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770424    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770427    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770429    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770432    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770434    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770437    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770439    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770442    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770447    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:44.774396 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770450    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770454    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770456    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770459    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770462    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770464    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770466    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770469    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770471    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770474    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770476    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770480    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770482    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770485    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770487    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770490    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770492    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770494    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770497    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770499    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:44.774848 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.770503    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770575    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770609    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770614    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770623    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770627    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770631    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770635    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770638    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770641    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770644    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770648    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770651    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770654    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770657    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770660    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770663    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770666    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770669    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770673    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770676    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770680    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770683    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770688    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 21:50:44.775326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770691    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770694    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770697    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770700    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770703    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770706    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770709    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770712    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770716    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770720    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770724    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770726    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770729    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770732    2582 flags.go:64] FLAG: --enable-server="true"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770735    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770740    2582 flags.go:64] FLAG: --event-burst="100"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770743    2582 flags.go:64] FLAG: --event-qps="50"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770746    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770749    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770751    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770755    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770758    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770761    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770764    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770767    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 21:50:44.775932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770770    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770772    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770775    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770778    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770781    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770783    2582 flags.go:64] FLAG: --feature-gates=""
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770788    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770792    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770795    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770798    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770801    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770803    2582 flags.go:64] FLAG: --help="false"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770806    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-142-209.ec2.internal"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770809    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770812    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770816    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770819    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770822    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770825    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770828    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770831    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770834    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770837    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770840    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 21:50:44.776585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770843    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770846    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770848    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770852    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770855    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770858    2582 flags.go:64] FLAG: --lock-file=""
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770860    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770863    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770866    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770871    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770874    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770877    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770879    2582 flags.go:64] FLAG: --logging-format="text"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770882    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770888    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770890    2582 flags.go:64] FLAG: --manifest-url=""
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770893    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770897    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770900    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770904    2582 flags.go:64] FLAG: --max-pods="110"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770907    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770910    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770913    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770917    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770920    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 21:50:44.777196 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770923    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770927    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770933    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770936    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770939    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770955    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770958    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770964    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770967    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770970    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770973    2582 flags.go:64] FLAG: --port="10250"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770976    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770979    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c1c787e9fd88795f"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770982    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770985    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770988    2582 flags.go:64] FLAG: --register-node="true"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770990    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770994    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770997    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.770999    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771002    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771007    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771010    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771012    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771015    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 21:50:44.777801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771018    2582 flags.go:64] FLAG: --runonce="false"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771021    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771024    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771027    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771030    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771032    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771036    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771039    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771042    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771045    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771048    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771051    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771053    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771056    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771060    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771062    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771067    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771071    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771073    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771078    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771081    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771083    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771086    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771089    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771092    2582 flags.go:64] FLAG: --v="2"
Sep 18 21:50:44.778544 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771095    2582 flags.go:64] FLAG: --version="false"
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771099    2582 flags.go:64] FLAG: --vmodule=""
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771102    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771107    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771211    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771216    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771219    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771221    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771224    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771227    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771229    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771232    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771235    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771240    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771243    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771246    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771249    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771251    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771254    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771256    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:44.779475 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771259    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771262    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771265    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771267    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771270    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771272    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771275    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771278    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771280    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771283    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771286    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771288    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771290    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771293    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771295    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771299    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771301    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771304    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771306    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771309    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:44.780039 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771311    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771314    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771316    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771318    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771321    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771325    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771327    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771330    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771332    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771335    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771337    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771340    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771342    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771345    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771347    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771350    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771353    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771356    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771358    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:44.780535 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771361    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771365    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771368    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771371    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771374    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771377    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771379    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771382    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771387    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771390    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771393    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771396    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771399    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771403    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771405    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771408    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771411    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771413    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771417    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:44.781010 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771420    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771422    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771425    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771427    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771430    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771433    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771435    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771437    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771440    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771443    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771446    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771449    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771452    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771455    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771457    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771460    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771462    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771465    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771468    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771470    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:44.781541 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771473    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771476    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771479    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771481    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.771483    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.771489    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.779912    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.779926    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.779960    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.779967    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780023    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780028    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780032    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780035    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:44.782035 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780038    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780041    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780044    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780047    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780049    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780052    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780055    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780058    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780060    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780063    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780066    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780068    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780071    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780074    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780076    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780079    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780082    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780086    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780090    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780093    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:44.782381 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780096    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780100    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780104    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780107    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780111    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780114    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780117    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780120    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780123    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780125    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780128    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780131    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780134    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780138    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780141    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780144    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780147    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780150    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780152    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:44.782854 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780155    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780158    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780160    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780163    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780166    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780168    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780171    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780174    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780176    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780179    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780181    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780184    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780187    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780190    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780192    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780195    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780197    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780200    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780203    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:44.783349 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780206    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780209    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780212    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780215    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780218    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780220    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780223    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780226    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780228    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780231    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780234    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780236    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780239    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780241    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780244    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780247    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780250    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780253    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780256    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780259    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:44.783789 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780261    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780264    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780266    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780270    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780272    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780275    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780277    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780280    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780283    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780285    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780288    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780290    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780293    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780296    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780298    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780301    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:44.784450 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780304    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.780308    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780421    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780426    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780429    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780432    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780435    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780438    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780440    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780443    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780445    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780448    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780450    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780453    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780456    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 21:50:44.784831 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780459    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780462    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780464    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780467    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780470    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780472    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780476    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780480    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780483    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780486    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780490    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780493    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780496    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780499    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780503    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780507    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780510    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780513    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780516    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 21:50:44.785222 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780518    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780521    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780524    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780527    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780529    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780532    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780534    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780537    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780539    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780542    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780545    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780548    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780551    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780553    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780556    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780559    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780561    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780564    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780566    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780569    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 21:50:44.785714 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780571    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780574    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780576    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780579    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780581    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780583    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780586    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780589    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780591    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780594    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780596    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780599    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780601    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780604    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780606    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780609    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780611    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780614    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780616    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780619    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 21:50:44.786216 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780621    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780624    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780627    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780629    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780632    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780634    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780637    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780640    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780642    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780645    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780647    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780650    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780652    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780655    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780657    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780660    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780662    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780665    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780667    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780670    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 21:50:44.786700 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780673    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780675    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780678    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780680    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780683    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780685    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:44.780688    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.780693    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.780797    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.783051    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.783793    2582 server.go:1019] "Starting client certificate rotation"
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.783895    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.783927    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 21:50:44.787246 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.786036    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:50:44.787565 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.787127    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 21:50:44.795592 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.795573    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 18 21:50:44.797174 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.797159    2582 log.go:25] "Validated CRI v1 image API"
Sep 18 21:50:44.798058 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.798043    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 21:50:44.803020 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.802997    2582 fs.go:135] Filesystem UUIDs: map[7025f83f-506b-4a3e-b8ab-a12d79b04fea:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e262905c-2428-4664-bc70-ed51e17460cf:/dev/nvme0n1p3]
Sep 18 21:50:44.803076 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.803020    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 21:50:44.808162 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.808057    2582 manager.go:216] Machine: {Timestamp:2026-09-18 21:50:44.806855174 +0000 UTC m=+1.840034058 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3189037 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24311542449972d69a89f34e465ece SystemUUID:ec243115-4244-9972-d69a-89f34e465ece BootID:281f172a-3440-47b0-abcc-c81085ffe3a2 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:a8:80:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:a8:80:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:0f:d3:3b:19:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 21:50:44.808162 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.808159    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 21:50:44.808269 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.808261    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 21:50:44.809622 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.809597    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 21:50:44.809748 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.809624    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 21:50:44.810728 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.810718    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 21:50:44.810758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.810731    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 21:50:44.810758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.810738    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:50:44.810758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.810748    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 21:50:44.810836 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.810810    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 21:50:44.814677 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.814666    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:50:44.814783 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.814774    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 21:50:44.814811 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.814806    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 21:50:44.814837 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.814814    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 21:50:44.814837 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.814825    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 21:50:44.814837 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.814835    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 18 21:50:44.814918 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.814843    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 21:50:44.815186 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.815165    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:50:44.815186 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.815183    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 21:50:44.820769 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.820745    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 21:50:44.823164 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.823148    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 21:50:44.823243 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.823177    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 21:50:44.824170 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824159    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 21:50:44.824206 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824176    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 21:50:44.824206 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824182    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824187    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824219    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824225    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824239    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824244    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824250    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824256    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 21:50:44.824264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824264    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 21:50:44.824460 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824269    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 21:50:44.824460 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824296    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 21:50:44.824460 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.824301    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 21:50:44.828055 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.828038    2582 server.go:1268] "Started kubelet"
Sep 18 21:50:44.828135 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.828077    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 21:50:44.828541 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.828127    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 21:50:44.828622 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.828559    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 21:50:44.828849 ip-10-0-142-209 systemd[1]: Started Kubernetes Kubelet.
Sep 18 21:50:44.829449 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.829133    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 21:50:44.829831 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.829775    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 21:50:44.832518 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.832490    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 21:50:44.832624 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.832547    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 21:50:44.832624 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.832584    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 21:50:44.834561 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.834538    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 21:50:44.834844 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.834829    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:50:44.836735 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.836717    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 21:50:44.836819 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.836740    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 21:50:44.836896 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.836879    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:44.837264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837247    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 21:50:44.837576 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837559    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 21:50:44.837668 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837504    2582 factory.go:55] Registering systemd factory
Sep 18 21:50:44.837668 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837611    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 18 21:50:44.837775 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837698    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 21:50:44.837775 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837727    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 21:50:44.837775 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837354    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 21:50:44.837966 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837933    2582 factory.go:153] Registering CRI-O factory
Sep 18 21:50:44.837966 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.837963    2582 factory.go:223] Registration of the crio container factory successfully
Sep 18 21:50:44.838095 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.838030    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 21:50:44.838095 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.838047    2582 factory.go:103] Registering Raw factory
Sep 18 21:50:44.838095 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.838058    2582 manager.go:1196] Started watching for new ooms in manager
Sep 18 21:50:44.838524 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.838459    2582 manager.go:349] Starting recovery of all containers
Sep 18 21:50:44.841061 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.841035    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 21:50:44.841187 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.841169    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 21:50:44.861638 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.861613    2582 manager.go:354] Recovery completed
Sep 18 21:50:44.861997 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.861978    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l7zwh"
Sep 18 21:50:44.867028 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.867014    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:44.867460 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.866483    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-209.ec2.internal.18d688bc6f3f6e39  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-209.ec2.internal,UID:ip-10-0-142-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-209.ec2.internal,},FirstTimestamp:2026-09-18 21:50:44.828007993 +0000 UTC m=+1.861186878,LastTimestamp:2026-09-18 21:50:44.828007993 +0000 UTC m=+1.861186878,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-209.ec2.internal,}"
Sep 18 21:50:44.868831 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.868807    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l7zwh"
Sep 18 21:50:44.868900 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.868862    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:44.868959 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.868902    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:44.868959 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.868914    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:44.869550 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.869535    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 21:50:44.869550 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.869548    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 21:50:44.869708 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.869562    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 21:50:44.869708 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.869575    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:44.870231 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.870217    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:44.870309 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.870243    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:44.870309 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.870253    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:44.872299 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.872285    2582 policy_none.go:49] "None policy: Start"
Sep 18 21:50:44.872374 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.872302    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 21:50:44.872374 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.872312    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 21:50:44.874273 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.874262    2582 policy_none.go:47] "Start"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.910901    2582 manager.go:337] "Starting Device Plugin manager"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.911067    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.911080    2582 server.go:87] "Starting device plugin registration server"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.911346    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.911369    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.911463    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.911551    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.911560    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.912032    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 21:50:44.918606 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.912081    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:44.966531 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.966505    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 21:50:44.967648 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.967635    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 21:50:44.967711 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.967676    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 21:50:44.967711 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.967699    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 21:50:44.967775 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:44.967741    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 21:50:44.972597 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:44.972577    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:45.011711 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.011683    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:45.012576 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.012555    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:45.012684 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.012589    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:45.012684 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.012604    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:45.012684 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.012633    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.030024 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.029975    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.030024 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.030000    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-209.ec2.internal\": node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.068673 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.068643    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal"]
Sep 18 21:50:45.068800 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.068712    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:45.068800 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.068769    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.069531 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.069516    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:45.069595 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.069545    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:45.069595 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.069555    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:45.071878 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.071866    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:45.071977 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.071964    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.072032 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.071991    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:45.072693 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.072676    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:45.072784 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.072683    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:45.072784 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.072725    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:45.072784 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.072736    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:45.072784 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.072701    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:45.072994 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.072783    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:45.074968 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.074954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.075031 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.074978    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 21:50:45.075624 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.075610    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 21:50:45.075704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.075636    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 21:50:45.075704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.075649    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeHasSufficientPID"
Sep 18 21:50:45.108493 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.108472    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-209.ec2.internal\" not found" node="ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.111774 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.111758    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-209.ec2.internal\" not found" node="ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.139258 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.139239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcfe32fe1636afad0e7829108abf9e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal\" (UID: \"bcfe32fe1636afad0e7829108abf9e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.139322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.139262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcfe32fe1636afad0e7829108abf9e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal\" (UID: \"bcfe32fe1636afad0e7829108abf9e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.139322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.139278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e9479c83aec32841bba93774c695096-config\") pod \"kube-apiserver-proxy-ip-10-0-142-209.ec2.internal\" (UID: \"6e9479c83aec32841bba93774c695096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.169015 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.168991    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.239665 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.239642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcfe32fe1636afad0e7829108abf9e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal\" (UID: \"bcfe32fe1636afad0e7829108abf9e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.239734 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.239672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcfe32fe1636afad0e7829108abf9e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal\" (UID: \"bcfe32fe1636afad0e7829108abf9e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.239734 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.239687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e9479c83aec32841bba93774c695096-config\") pod \"kube-apiserver-proxy-ip-10-0-142-209.ec2.internal\" (UID: \"6e9479c83aec32841bba93774c695096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.239734 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.239713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6e9479c83aec32841bba93774c695096-config\") pod \"kube-apiserver-proxy-ip-10-0-142-209.ec2.internal\" (UID: \"6e9479c83aec32841bba93774c695096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.239833 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.239730    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bcfe32fe1636afad0e7829108abf9e4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal\" (UID: \"bcfe32fe1636afad0e7829108abf9e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.239833 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.239735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcfe32fe1636afad0e7829108abf9e4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal\" (UID: \"bcfe32fe1636afad0e7829108abf9e4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.269581 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.269563    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.369891 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.369860    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.412054 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.412031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.415170 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.415155    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:45.470797 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.470772    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.571241 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.571220    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.671805 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.671731    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.687260 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.687233    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:45.772509 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.772478    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.783634 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.783613    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 21:50:45.783755 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.783736    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:50:45.783794 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.783772    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 21:50:45.873496 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.873467    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.874580 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.874544    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 21:45:44 +0000 UTC" deadline="2028-04-10 11:14:53.628953685 +0000 UTC"
Sep 18 21:50:45.874641 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.874582    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13669h24m7.75437651s"
Sep 18 21:50:45.974231 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:45.974210    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-209.ec2.internal\" not found"
Sep 18 21:50:45.976090 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.976076    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:45.991775 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:45.991743    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfe32fe1636afad0e7829108abf9e4d.slice/crio-830b62e7774ad42257ef9553d6892ca012719f52919b8345852166a33c608492 WatchSource:0}: Error finding container 830b62e7774ad42257ef9553d6892ca012719f52919b8345852166a33c608492: Status 404 returned error can't find the container with id 830b62e7774ad42257ef9553d6892ca012719f52919b8345852166a33c608492
Sep 18 21:50:45.992292 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:45.992278    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9479c83aec32841bba93774c695096.slice/crio-bb7b8583a010fdc53a7300021f837982acf76e72b41d823a3866229b17a9d420 WatchSource:0}: Error finding container bb7b8583a010fdc53a7300021f837982acf76e72b41d823a3866229b17a9d420: Status 404 returned error can't find the container with id bb7b8583a010fdc53a7300021f837982acf76e72b41d823a3866229b17a9d420
Sep 18 21:50:45.996468 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:45.996450    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:50:46.037896 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.037871    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:46.054986 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.054967    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:50:46.056493 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.056481    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal"
Sep 18 21:50:46.067592 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.067577    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 21:50:46.363176 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.363072    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:46.816396 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.816364    2582 apiserver.go:52] "Watching apiserver"
Sep 18 21:50:46.823009 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.822981    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 21:50:46.823432 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.823399    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal","openshift-cluster-node-tuning-operator/tuned-r26z5","openshift-image-registry/node-ca-c4kct","openshift-multus/network-metrics-daemon-nwtqb","openshift-network-diagnostics/network-check-target-x84ph","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt","openshift-dns/node-resolver-2wm8s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal","openshift-multus/multus-76p9s","openshift-multus/multus-additional-cni-plugins-fn6vl","openshift-network-operator/iptables-alerter-7mx7p","openshift-ovn-kubernetes/ovnkube-node-rmj6f","kube-system/global-pull-secret-syncer-gddqh","kube-system/konnectivity-agent-976mf"]
Sep 18 21:50:46.826343 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.826323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.828531 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.828510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.828626 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.828583    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:50:46.829045 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.829023    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 21:50:46.829207 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.829153    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xq7lr\""
Sep 18 21:50:46.829312 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.829218    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 21:50:46.831668 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.831292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.833846 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.833826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:46.833967 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.833905    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:50:46.835134 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.835027    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 21:50:46.835235 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.835216    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 21:50:46.835325 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.835241    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 21:50:46.835392 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.835343    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g6tcr\""
Sep 18 21:50:46.836189 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.835898    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:46.836189 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.835992    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:46.838161 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.838141    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 21:50:46.839597 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.839579    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.842018 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.841826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.842117 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.842065    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 21:50:46.842285 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.842245    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 21:50:46.842285 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.842278    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 21:50:46.842481 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.842460    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4g64n\""
Sep 18 21:50:46.844451 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.844415    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:46.844764 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.844742    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 21:50:46.844972 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.844955    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 21:50:46.845057 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.844970    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 21:50:46.845057 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.845010    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 21:50:46.845057 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.845042    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4m67k\""
Sep 18 21:50:46.846498 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.846477    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 21:50:46.846844 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.846826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-999qd\""
Sep 18 21:50:46.847161 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.847144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.848156 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmmd\" (UniqueName: \"kubernetes.io/projected/52d1076e-4c86-496e-a462-154feab9b8b9-kube-api-access-hkmmd\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:46.848250 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:46.848250 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.848250 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e2d73b3-52ef-4f05-839a-051bab482a3c-host\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.848408 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848270    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e2d73b3-52ef-4f05-839a-051bab482a3c-serviceca\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.848408 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-registration-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.848408 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/310ea0c0-e1ce-4eb1-871f-33ef46f48827-hosts-file\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.848561 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310ea0c0-e1ce-4eb1-871f-33ef46f48827-tmp-dir\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-dbus\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.848993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6q4\" (UniqueName: \"kubernetes.io/projected/6e2d73b3-52ef-4f05-839a-051bab482a3c-kube-api-access-mh6q4\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-device-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-sys-fs\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65257\" (UniqueName: \"kubernetes.io/projected/310ea0c0-e1ce-4eb1-871f-33ef46f48827-kube-api-access-65257\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-kubelet-config\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-socket-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqdf\" (UniqueName: \"kubernetes.io/projected/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-kube-api-access-khqdf\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.849401 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.850740 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.849982    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:50:46.850740 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.850051    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 21:50:46.850740 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.850255    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5h7cd\""
Sep 18 21:50:46.850740 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.850300    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 21:50:46.852834 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.852572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.855165 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.854844    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:46.855165 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.855018    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-txkr2\""
Sep 18 21:50:46.855455 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.855435    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 21:50:46.855546 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.855468    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 21:50:46.857350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.857278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.857436 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.857377    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 21:50:46.857691 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.857526    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:50:46.857979 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.857960    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 21:50:46.858082 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.857985    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bxpfh\""
Sep 18 21:50:46.858859 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.858841    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 21:50:46.861257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.860836    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 21:50:46.861257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.860858    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 21:50:46.861257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.860892    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 21:50:46.861257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.861009    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 21:50:46.861257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.861110    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lp4hd\""
Sep 18 21:50:46.861257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.861123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 21:50:46.861257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.861150    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 21:50:46.891356 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.891334    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fml82"
Sep 18 21:50:46.897681 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.897661    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fml82"
Sep 18 21:50:46.939352 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.939324    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 21:50:46.950044 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6q4\" (UniqueName: \"kubernetes.io/projected/6e2d73b3-52ef-4f05-839a-051bab482a3c-kube-api-access-mh6q4\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.950198 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-device-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.950198 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.950198 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-tmp\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.950198 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mq2j\" (UniqueName: \"kubernetes.io/projected/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-kube-api-access-6mq2j\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.950198 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.950198 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-device-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.950478 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-lib-modules\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.950478 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.950478 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmmd\" (UniqueName: \"kubernetes.io/projected/52d1076e-4c86-496e-a462-154feab9b8b9-kube-api-access-hkmmd\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:46.950478 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.950478 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.950380    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:46.950478 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-kubelet\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.950521    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret podName:299a1b6f-acaa-4fe6-a60f-0ef9f999970f nodeName:}" failed. No retries permitted until 2026-09-18 21:50:47.450496478 +0000 UTC m=+4.483675349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret") pod "global-pull-secret-syncer-gddqh" (UID: "299a1b6f-acaa-4fe6-a60f-0ef9f999970f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-var-lib-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-log-socket\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysconfig\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/310ea0c0-e1ce-4eb1-871f-33ef46f48827-hosts-file\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysctl-d\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950675    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-run\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/310ea0c0-e1ce-4eb1-871f-33ef46f48827-hosts-file\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-var-lib-kubelet\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.950758 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-netns\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-cni-bin\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-conf-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-dbus\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-sys-fs\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovn-node-metrics-cert\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950901    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovnkube-script-lib\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b132a4a-79f6-4d95-a440-7962162b5a24-agent-certs\") pod \"konnectivity-agent-976mf\" (UID: \"8b132a4a-79f6-4d95-a440-7962162b5a24\") " pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.950987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-iptables-alerter-script\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-host-slash\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-tuned\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-kubelet-config\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-dbus\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hzjp\" (UniqueName: \"kubernetes.io/projected/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-kube-api-access-7hzjp\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-hostroot\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-multus-certs\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-cni-bin\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.951386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951221    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-kubelet-config\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-cni-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-cni-multus\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-sys-fs\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9v2\" (UniqueName: \"kubernetes.io/projected/7c8563dd-0e2e-4aab-8814-c44cce27c793-kube-api-access-bk9v2\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9ww\" (UniqueName: \"kubernetes.io/projected/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-kube-api-access-9h9ww\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e2d73b3-52ef-4f05-839a-051bab482a3c-host\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6e2d73b3-52ef-4f05-839a-051bab482a3c-host\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e2d73b3-52ef-4f05-839a-051bab482a3c-serviceca\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-system-cni-dir\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-node-log\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-env-overrides\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-system-cni-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-systemd-units\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-slash\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.952245 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-cnibin\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-daemon-config\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovnkube-config\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysctl-conf\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqmx\" (UniqueName: \"kubernetes.io/projected/847de099-1e25-441a-8dff-e4d9378896bd-kube-api-access-bvqmx\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-kubelet\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6e2d73b3-52ef-4f05-839a-051bab482a3c-serviceca\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-registration-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.951993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-cni-netd\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-cnibin\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c8563dd-0e2e-4aab-8814-c44cce27c793-cni-binary-copy\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65257\" (UniqueName: \"kubernetes.io/projected/310ea0c0-e1ce-4eb1-871f-33ef46f48827-kube-api-access-65257\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-ovn\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-socket-dir-parent\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-socket-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.953046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqdf\" (UniqueName: \"kubernetes.io/projected/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-kube-api-access-khqdf\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-systemd\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-etc-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-kubernetes\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-systemd\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-run-netns\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-modprobe-d\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-sys\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952496    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-etc-kubernetes\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310ea0c0-e1ce-4eb1-871f-33ef46f48827-tmp-dir\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.952497    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-registration-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.952597    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:47.452580121 +0000 UTC m=+4.485758998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-host\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:46.953704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:46.954322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-k8s-cni-cncf-io\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:46.954322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952719    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b132a4a-79f6-4d95-a440-7962162b5a24-konnectivity-ca\") pod \"konnectivity-agent-976mf\" (UID: \"8b132a4a-79f6-4d95-a440-7962162b5a24\") " pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:46.954322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-socket-dir\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.954322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.952851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-etc-selinux\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.954322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.953047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/310ea0c0-e1ce-4eb1-871f-33ef46f48827-tmp-dir\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.966327 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.966300    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 21:50:46.970337 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.970280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmmd\" (UniqueName: \"kubernetes.io/projected/52d1076e-4c86-496e-a462-154feab9b8b9-kube-api-access-hkmmd\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:46.976606 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.972999    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:46.976606 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.973030    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:46.976606 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.973043    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lrpfp for pod openshift-network-diagnostics/network-check-target-x84ph: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:46.976606 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:46.973108    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp podName:d421ac05-a1cc-4336-b61b-bf5228b0573d nodeName:}" failed. No retries permitted until 2026-09-18 21:50:47.473091143 +0000 UTC m=+4.506270030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lrpfp" (UniqueName: "kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp") pod "network-check-target-x84ph" (UID: "d421ac05-a1cc-4336-b61b-bf5228b0573d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:46.976606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.976120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65257\" (UniqueName: \"kubernetes.io/projected/310ea0c0-e1ce-4eb1-871f-33ef46f48827-kube-api-access-65257\") pod \"node-resolver-2wm8s\" (UID: \"310ea0c0-e1ce-4eb1-871f-33ef46f48827\") " pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:46.978456 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.978430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6q4\" (UniqueName: \"kubernetes.io/projected/6e2d73b3-52ef-4f05-839a-051bab482a3c-kube-api-access-mh6q4\") pod \"node-ca-c4kct\" (UID: \"6e2d73b3-52ef-4f05-839a-051bab482a3c\") " pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:46.979167 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.978625    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqdf\" (UniqueName: \"kubernetes.io/projected/0caea700-b5e5-42c9-97ec-66c0c5bf19b8-kube-api-access-khqdf\") pod \"aws-ebs-csi-driver-node-zg8wt\" (UID: \"0caea700-b5e5-42c9-97ec-66c0c5bf19b8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:46.988669 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.988624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal" event={"ID":"bcfe32fe1636afad0e7829108abf9e4d","Type":"ContainerStarted","Data":"830b62e7774ad42257ef9553d6892ca012719f52919b8345852166a33c608492"}
Sep 18 21:50:46.989881 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:46.989855    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal" event={"ID":"6e9479c83aec32841bba93774c695096","Type":"ContainerStarted","Data":"bb7b8583a010fdc53a7300021f837982acf76e72b41d823a3866229b17a9d420"}
Sep 18 21:50:47.053669 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-system-cni-dir\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.053669 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-node-log\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-env-overrides\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053725    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-system-cni-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053759    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-system-cni-dir\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-system-cni-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053766    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-node-log\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-systemd-units\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.053920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-slash\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053959    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-cnibin\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-systemd-units\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-daemon-config\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-cnibin\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.053975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-slash\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovnkube-config\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysctl-conf\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqmx\" (UniqueName: \"kubernetes.io/projected/847de099-1e25-441a-8dff-e4d9378896bd-kube-api-access-bvqmx\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-kubelet\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-cni-netd\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-cnibin\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c8563dd-0e2e-4aab-8814-c44cce27c793-cni-binary-copy\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054217    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-ovn\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysctl-conf\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.054377 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-socket-dir-parent\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-socket-dir-parent\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-systemd\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-etc-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-kubernetes\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-systemd\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-env-overrides\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-run-netns\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-modprobe-d\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054433    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-cni-netd\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-sys\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-etc-kubernetes\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-ovn\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-cnibin\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-host\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-kubelet\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-run-netns\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.055181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-run-ovn-kubernetes\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-k8s-cni-cncf-io\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-systemd\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b132a4a-79f6-4d95-a440-7962162b5a24-konnectivity-ca\") pod \"konnectivity-agent-976mf\" (UID: \"8b132a4a-79f6-4d95-a440-7962162b5a24\") " pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-daemon-config\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-kubernetes\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-etc-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-systemd\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovnkube-config\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-etc-kubernetes\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-run-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-tmp\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mq2j\" (UniqueName: \"kubernetes.io/projected/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-kube-api-access-6mq2j\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054672    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-sys\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054692    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-lib-modules\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056096 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-kubelet\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-var-lib-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-modprobe-d\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-log-socket\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/847de099-1e25-441a-8dff-e4d9378896bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysconfig\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-log-socket\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysctl-d\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-run\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-var-lib-kubelet\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-netns\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysconfig\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-cni-bin\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.054986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-conf-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovn-node-metrics-cert\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovnkube-script-lib\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.056899 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055066    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c8563dd-0e2e-4aab-8814-c44cce27c793-cni-binary-copy\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b132a4a-79f6-4d95-a440-7962162b5a24-agent-certs\") pod \"konnectivity-agent-976mf\" (UID: \"8b132a4a-79f6-4d95-a440-7962162b5a24\") " pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-iptables-alerter-script\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-sysctl-d\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-host-slash\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-tuned\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-run\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzjp\" (UniqueName: \"kubernetes.io/projected/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-kube-api-access-7hzjp\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-hostroot\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-multus-certs\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-cni-bin\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055301    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b132a4a-79f6-4d95-a440-7962162b5a24-konnectivity-ca\") pod \"konnectivity-agent-976mf\" (UID: \"8b132a4a-79f6-4d95-a440-7962162b5a24\") " pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-lib-modules\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-host-slash\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-cni-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055372    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-kubelet\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-host\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.057649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-cni-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-multus-conf-dir\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-cni-multus\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9v2\" (UniqueName: \"kubernetes.io/projected/7c8563dd-0e2e-4aab-8814-c44cce27c793-kube-api-access-bk9v2\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9ww\" (UniqueName: \"kubernetes.io/projected/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-kube-api-access-9h9ww\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055439    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-var-lib-openvswitch\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-hostroot\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055698    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-cni-multus\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-iptables-alerter-script\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055796    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-k8s-cni-cncf-io\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-netns\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055021    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-var-lib-kubelet\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-var-lib-cni-bin\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c8563dd-0e2e-4aab-8814-c44cce27c793-host-run-multus-certs\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.055958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/847de099-1e25-441a-8dff-e4d9378896bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.056043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-host-cni-bin\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.056397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovnkube-script-lib\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.057908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-tmp\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.058413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.057969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-etc-tuned\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.059131 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.058070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b132a4a-79f6-4d95-a440-7962162b5a24-agent-certs\") pod \"konnectivity-agent-976mf\" (UID: \"8b132a4a-79f6-4d95-a440-7962162b5a24\") " pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:47.059131 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.058336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-ovn-node-metrics-cert\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.068324 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.068274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqmx\" (UniqueName: \"kubernetes.io/projected/847de099-1e25-441a-8dff-e4d9378896bd-kube-api-access-bvqmx\") pod \"multus-additional-cni-plugins-fn6vl\" (UID: \"847de099-1e25-441a-8dff-e4d9378896bd\") " pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.076145 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.076124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzjp\" (UniqueName: \"kubernetes.io/projected/8e53a3c0-3b7d-4d05-9357-c7d53f7d096b-kube-api-access-7hzjp\") pod \"ovnkube-node-rmj6f\" (UID: \"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b\") " pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.076677 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.076653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9ww\" (UniqueName: \"kubernetes.io/projected/8b5c6a52-d31b-474f-b791-8e5f9b3254fe-kube-api-access-9h9ww\") pod \"iptables-alerter-7mx7p\" (UID: \"8b5c6a52-d31b-474f-b791-8e5f9b3254fe\") " pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:47.079373 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.079353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mq2j\" (UniqueName: \"kubernetes.io/projected/89b8e808-fc74-4f6e-9eae-40876d8ed2ba-kube-api-access-6mq2j\") pod \"tuned-r26z5\" (UID: \"89b8e808-fc74-4f6e-9eae-40876d8ed2ba\") " pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.085240 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.085222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9v2\" (UniqueName: \"kubernetes.io/projected/7c8563dd-0e2e-4aab-8814-c44cce27c793-kube-api-access-bk9v2\") pod \"multus-76p9s\" (UID: \"7c8563dd-0e2e-4aab-8814-c44cce27c793\") " pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.138518 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.138488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2wm8s"
Sep 18 21:50:47.147029 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.147010    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4kct"
Sep 18 21:50:47.157433 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.157415    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt"
Sep 18 21:50:47.164186 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.164160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76p9s"
Sep 18 21:50:47.170812 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.170790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:50:47.178425 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.178403    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r26z5"
Sep 18 21:50:47.185020 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.185001    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fn6vl"
Sep 18 21:50:47.194722 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.194705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7mx7p"
Sep 18 21:50:47.199215 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.199199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:50:47.350934 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.350852    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:47.458833 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.458800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:47.459015 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.458853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:47.459015 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.458967    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:47.459015 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.459000    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:47.459153 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.459043    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret podName:299a1b6f-acaa-4fe6-a60f-0ef9f999970f nodeName:}" failed. No retries permitted until 2026-09-18 21:50:48.459021072 +0000 UTC m=+5.492199943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret") pod "global-pull-secret-syncer-gddqh" (UID: "299a1b6f-acaa-4fe6-a60f-0ef9f999970f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:47.459153 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.459064    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:48.459054361 +0000 UTC m=+5.492233238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:47.559956 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.559923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:47.560117 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.560053    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:47.560117 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.560073    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:47.560117 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.560086    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lrpfp for pod openshift-network-diagnostics/network-check-target-x84ph: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:47.560234 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.560145    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp podName:d421ac05-a1cc-4336-b61b-bf5228b0573d nodeName:}" failed. No retries permitted until 2026-09-18 21:50:48.560130208 +0000 UTC m=+5.593309081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrpfp" (UniqueName: "kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp") pod "network-check-target-x84ph" (UID: "d421ac05-a1cc-4336-b61b-bf5228b0573d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:47.884612 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:47.884584    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e53a3c0_3b7d_4d05_9357_c7d53f7d096b.slice/crio-0b315a71dea8d37f890ccad0c76c51200200a63e9982a9d00f700a46c504fc5b WatchSource:0}: Error finding container 0b315a71dea8d37f890ccad0c76c51200200a63e9982a9d00f700a46c504fc5b: Status 404 returned error can't find the container with id 0b315a71dea8d37f890ccad0c76c51200200a63e9982a9d00f700a46c504fc5b
Sep 18 21:50:47.889433 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:50:47.889403    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5c6a52_d31b_474f_b791_8e5f9b3254fe.slice/crio-69f35e98445818af1934d371b9150076703ae3d53a656949f0d149fff65b2a81 WatchSource:0}: Error finding container 69f35e98445818af1934d371b9150076703ae3d53a656949f0d149fff65b2a81: Status 404 returned error can't find the container with id 69f35e98445818af1934d371b9150076703ae3d53a656949f0d149fff65b2a81
Sep 18 21:50:47.901580 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.901532    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:45:46 +0000 UTC" deadline="2028-04-28 00:49:50.169407721 +0000 UTC"
Sep 18 21:50:47.901646 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.901582    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14090h59m2.267829887s"
Sep 18 21:50:47.968875 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.968810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:47.968875 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.968861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:47.969035 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.968975    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:50:47.969088 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:47.969030    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:47.995614 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.995589    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"0b315a71dea8d37f890ccad0c76c51200200a63e9982a9d00f700a46c504fc5b"}
Sep 18 21:50:47.998161 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.998138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerStarted","Data":"a4d8f9b8f06aecc39b5dae10cbe060b508c2a98bf51fd6bbec736070c1c7ade7"}
Sep 18 21:50:47.999291 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:47.999272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt" event={"ID":"0caea700-b5e5-42c9-97ec-66c0c5bf19b8","Type":"ContainerStarted","Data":"db0061968902baca635b4a82ab26171a24c206ef9088b197bb3b9c49340328aa"}
Sep 18 21:50:48.003673 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.003651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal" event={"ID":"6e9479c83aec32841bba93774c695096","Type":"ContainerStarted","Data":"b90830fdb2889da934c92ca02d13ab69b05d91e393b907e6cd71a20e102c3428"}
Sep 18 21:50:48.005597 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.005572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wm8s" event={"ID":"310ea0c0-e1ce-4eb1-871f-33ef46f48827","Type":"ContainerStarted","Data":"cf2de7c70ec86ec4c71c6e16e874ed6b24e48fe42576c816ccd74782611983e5"}
Sep 18 21:50:48.006725 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.006708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7mx7p" event={"ID":"8b5c6a52-d31b-474f-b791-8e5f9b3254fe","Type":"ContainerStarted","Data":"69f35e98445818af1934d371b9150076703ae3d53a656949f0d149fff65b2a81"}
Sep 18 21:50:48.007991 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.007968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4kct" event={"ID":"6e2d73b3-52ef-4f05-839a-051bab482a3c","Type":"ContainerStarted","Data":"95733f6f1b14131c1f395ccabbea1524c4331169cb4595341dae31ac392234f1"}
Sep 18 21:50:48.010561 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.010541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r26z5" event={"ID":"89b8e808-fc74-4f6e-9eae-40876d8ed2ba","Type":"ContainerStarted","Data":"7484f37b005d502679d9d16e3d395435c280b336851e52b1c7b5dfdf5fcb80e0"}
Sep 18 21:50:48.011731 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.011711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76p9s" event={"ID":"7c8563dd-0e2e-4aab-8814-c44cce27c793","Type":"ContainerStarted","Data":"dfc2cdb6bb9b7f8f4194139c301a9dcd392efa187f327f8c683ff26350cdddea"}
Sep 18 21:50:48.015485 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.015467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-976mf" event={"ID":"8b132a4a-79f6-4d95-a440-7962162b5a24","Type":"ContainerStarted","Data":"b70e07d9e7e5724bfa50410b468367fcaa358251bddab84ea54369a3a801a40b"}
Sep 18 21:50:48.026524 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.026475    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal" podStartSLOduration=2.026459053 podStartE2EDuration="2.026459053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:48.025866337 +0000 UTC m=+5.059045232" watchObservedRunningTime="2026-09-18 21:50:48.026459053 +0000 UTC m=+5.059637949"
Sep 18 21:50:48.468086 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.467580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:48.468086 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.467889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:48.468086 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.468037    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:48.468358 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.468099    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:50.468082225 +0000 UTC m=+7.501261103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:48.468490 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.468471    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:48.468567 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.468526    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret podName:299a1b6f-acaa-4fe6-a60f-0ef9f999970f nodeName:}" failed. No retries permitted until 2026-09-18 21:50:50.468511279 +0000 UTC m=+7.501690166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret") pod "global-pull-secret-syncer-gddqh" (UID: "299a1b6f-acaa-4fe6-a60f-0ef9f999970f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:48.532849 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.532797    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 21:50:48.568701 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.568666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:48.568912 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.568863    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:48.568912 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.568883    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:48.568912 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.568895    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lrpfp for pod openshift-network-diagnostics/network-check-target-x84ph: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:48.569189 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.568965    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp podName:d421ac05-a1cc-4336-b61b-bf5228b0573d nodeName:}" failed. No retries permitted until 2026-09-18 21:50:50.56893261 +0000 UTC m=+7.602111497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrpfp" (UniqueName: "kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp") pod "network-check-target-x84ph" (UID: "d421ac05-a1cc-4336-b61b-bf5228b0573d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:48.902240 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.902194    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 21:45:46 +0000 UTC" deadline="2028-07-02 02:58:23.800466858 +0000 UTC"
Sep 18 21:50:48.902240 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.902235    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15653h7m34.898236401s"
Sep 18 21:50:48.968624 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:48.968581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:48.968802 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:48.968707    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:50:49.031358 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:49.031321    2582 generic.go:356] "Generic (PLEG): container finished" podID="bcfe32fe1636afad0e7829108abf9e4d" containerID="cc78378ee71c2da4b51ed4661e78b9c8cf42f13d090d5900e54559674bc4cdeb" exitCode=0
Sep 18 21:50:49.032302 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:49.032274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal" event={"ID":"bcfe32fe1636afad0e7829108abf9e4d","Type":"ContainerDied","Data":"cc78378ee71c2da4b51ed4661e78b9c8cf42f13d090d5900e54559674bc4cdeb"}
Sep 18 21:50:49.968612 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:49.968576    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:49.969148 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:49.968727    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:50:49.969148 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:49.968808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:49.969148 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:49.968878    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:50.052310 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:50.052271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal" event={"ID":"bcfe32fe1636afad0e7829108abf9e4d","Type":"ContainerStarted","Data":"805b167aa8ec5ab6f152659c61abfd50cda24859af1931197ffdd782aca74088"}
Sep 18 21:50:50.071240 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:50.071184    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-209.ec2.internal" podStartSLOduration=4.071163181 podStartE2EDuration="4.071163181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:50:50.070206852 +0000 UTC m=+7.103385746" watchObservedRunningTime="2026-09-18 21:50:50.071163181 +0000 UTC m=+7.104342076"
Sep 18 21:50:50.485042 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:50.484259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:50.485042 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:50.484382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:50.485042 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.484518    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:50.485042 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.484578    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret podName:299a1b6f-acaa-4fe6-a60f-0ef9f999970f nodeName:}" failed. No retries permitted until 2026-09-18 21:50:54.484560968 +0000 UTC m=+11.517739847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret") pod "global-pull-secret-syncer-gddqh" (UID: "299a1b6f-acaa-4fe6-a60f-0ef9f999970f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:50.485042 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.484964    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:50.485042 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.485009    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:50:54.484995813 +0000 UTC m=+11.518174688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:50.585359 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:50.585131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:50.585359 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.585299    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:50.585359 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.585320    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:50.585359 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.585332    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lrpfp for pod openshift-network-diagnostics/network-check-target-x84ph: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:50.585677 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.585395    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp podName:d421ac05-a1cc-4336-b61b-bf5228b0573d nodeName:}" failed. No retries permitted until 2026-09-18 21:50:54.585377667 +0000 UTC m=+11.618556544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrpfp" (UniqueName: "kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp") pod "network-check-target-x84ph" (UID: "d421ac05-a1cc-4336-b61b-bf5228b0573d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:50.969969 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:50.969317    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:50.969969 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:50.969452    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:50:51.968067 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:51.968033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:51.968246 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:51.968182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:50:51.968576 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:51.968553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:51.968687 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:51.968648    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:52.969088 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:52.968554    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:52.969088 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:52.968708    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:50:53.968668 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:53.968138    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:53.968668 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:53.968276    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:53.968668 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:53.968350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:53.968668 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:53.968448    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:50:54.519103 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:54.519066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:54.519559 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:54.519144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:54.519559 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.519302    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:54.519559 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.519364    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:02.519351464 +0000 UTC m=+19.552530336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:50:54.519559 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.519302    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:54.519559 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.519443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret podName:299a1b6f-acaa-4fe6-a60f-0ef9f999970f nodeName:}" failed. No retries permitted until 2026-09-18 21:51:02.519430782 +0000 UTC m=+19.552609673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret") pod "global-pull-secret-syncer-gddqh" (UID: "299a1b6f-acaa-4fe6-a60f-0ef9f999970f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:50:54.620117 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:54.620081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:54.620301 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.620290    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:50:54.620372 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.620307    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:50:54.620372 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.620320    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lrpfp for pod openshift-network-diagnostics/network-check-target-x84ph: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:54.620477 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.620381    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp podName:d421ac05-a1cc-4336-b61b-bf5228b0573d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:02.620362274 +0000 UTC m=+19.653541151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrpfp" (UniqueName: "kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp") pod "network-check-target-x84ph" (UID: "d421ac05-a1cc-4336-b61b-bf5228b0573d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:50:54.970088 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:54.970057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:54.970282 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:54.970163    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:50:55.968135 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:55.968107    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:55.968555 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:55.968107    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:55.968555 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:55.968242    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:50:55.968555 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:55.968295    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:56.968011 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:56.967979    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:56.968160 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:56.968117    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:50:57.968056 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:57.968024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:57.968225 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:57.968023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:57.968225 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:57.968164    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:57.968581 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:57.968250    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:50:58.968960 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:58.968914    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:50:58.969481 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:58.969080    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:50:59.968564 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:59.968530    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:50:59.968564 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:50:59.968579    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:50:59.968804 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:59.968654    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:50:59.968804 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:50:59.968755    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:00.968390 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:00.968360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:00.968745 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:00.968488    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:01.968026 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:01.968000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:01.968248 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:01.967999    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:01.968248 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:01.968139    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:01.968248 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:01.968198    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:02.578079 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:02.578039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:02.578531 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:02.578103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:02.578531 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.578222    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:02.578531 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.578294    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret podName:299a1b6f-acaa-4fe6-a60f-0ef9f999970f nodeName:}" failed. No retries permitted until 2026-09-18 21:51:18.578279008 +0000 UTC m=+35.611457883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret") pod "global-pull-secret-syncer-gddqh" (UID: "299a1b6f-acaa-4fe6-a60f-0ef9f999970f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:02.578531 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.578226    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:02.578531 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.578378    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:18.578360823 +0000 UTC m=+35.611539694 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:02.678873 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:02.678842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:02.679079 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.678982    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:02.679079 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.679004    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:02.679079 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.679016    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lrpfp for pod openshift-network-diagnostics/network-check-target-x84ph: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:02.679079 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.679073    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp podName:d421ac05-a1cc-4336-b61b-bf5228b0573d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:18.679059574 +0000 UTC m=+35.712238464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrpfp" (UniqueName: "kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp") pod "network-check-target-x84ph" (UID: "d421ac05-a1cc-4336-b61b-bf5228b0573d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:02.968883 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:02.968845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:02.969480 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:02.969317    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:03.968185 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:03.968152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:03.968185 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:03.968185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:03.968638 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:03.968281    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:03.968638 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:03.968426    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:04.969393 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:04.969364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:04.969771 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:04.969462    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:05.968742 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:05.968573    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:05.968918 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:05.968594    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:05.968918 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:05.968853    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:05.968918 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:05.968885    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:06.087295 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087137    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 21:51:06.087856 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087624    2582 generic.go:356] "Generic (PLEG): container finished" podID="8e53a3c0-3b7d-4d05-9357-c7d53f7d096b" containerID="a4c854cecf71f0c7582e50e2b62affc4dc826ab0e3b9fc91a9a6e252e74563b0" exitCode=1
Sep 18 21:51:06.087856 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"d6463a0b979364ba725397b5e41940e96de4b340cda572fc7db38198c682a32d"}
Sep 18 21:51:06.087856 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"70e94c7f2cc3be421f433bf20770357bf8b054e6d42f7c431a2a88f0dffc5381"}
Sep 18 21:51:06.087856 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"e9d8597857055aae073fa2558b1a30a3d5613edaadc7d0a3a3f61afa9f64a4f1"}
Sep 18 21:51:06.088063 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"baf65cb7794fba3733135e609388de720c2811a3dbd0bb0474673db4258299d7"}
Sep 18 21:51:06.088063 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087899    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerDied","Data":"a4c854cecf71f0c7582e50e2b62affc4dc826ab0e3b9fc91a9a6e252e74563b0"}
Sep 18 21:51:06.088063 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.087929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"3d09d3bbb6d0130a4705ad0c227bf26a4b06f3c768ff0d8726895cfd25d829b7"}
Sep 18 21:51:06.089331 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.089311    2582 generic.go:356] "Generic (PLEG): container finished" podID="847de099-1e25-441a-8dff-e4d9378896bd" containerID="ec53fb170e38853f9e18789cb895294a871537bd7c1dbbeec45c05804946a973" exitCode=0
Sep 18 21:51:06.089438 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.089386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerDied","Data":"ec53fb170e38853f9e18789cb895294a871537bd7c1dbbeec45c05804946a973"}
Sep 18 21:51:06.091002 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.090978    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt" event={"ID":"0caea700-b5e5-42c9-97ec-66c0c5bf19b8","Type":"ContainerStarted","Data":"719962d9e737594f3e18493cb9169faa147ff65bb8fc869f8c6f6a3d1838be20"}
Sep 18 21:51:06.092478 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.092457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2wm8s" event={"ID":"310ea0c0-e1ce-4eb1-871f-33ef46f48827","Type":"ContainerStarted","Data":"bcd660f6df66eeedb396a5f27af2c220b23e09832c82d7710d3f02cf96f09369"}
Sep 18 21:51:06.093877 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.093859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4kct" event={"ID":"6e2d73b3-52ef-4f05-839a-051bab482a3c","Type":"ContainerStarted","Data":"32b729004af2832a02426bb570886fdee3f862395daf7775a547fc053169982d"}
Sep 18 21:51:06.095180 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.095161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r26z5" event={"ID":"89b8e808-fc74-4f6e-9eae-40876d8ed2ba","Type":"ContainerStarted","Data":"27ce6a7e85853ddbdaea111bdf747e27e61dba5eaa872e49b3e1364a9939b0e9"}
Sep 18 21:51:06.096534 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.096515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76p9s" event={"ID":"7c8563dd-0e2e-4aab-8814-c44cce27c793","Type":"ContainerStarted","Data":"3af5480b3f1707714c688bf5a8421b1efe9986204dfdb500401f858fd0e23ab5"}
Sep 18 21:51:06.097989 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.097971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-976mf" event={"ID":"8b132a4a-79f6-4d95-a440-7962162b5a24","Type":"ContainerStarted","Data":"7849bf9a2d5ebfeeb2d310e71ad95338083261361c560e9c035292e83a85ebce"}
Sep 18 21:51:06.129805 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.129788    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 21:51:06.185696 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.185655    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-76p9s" podStartSLOduration=4.088320506 podStartE2EDuration="21.18564151s" totalImagesPullingTime="17.097321004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:06.159268671 +0000 UTC m=+23.192447564" watchObservedRunningTime="2026-09-18 21:51:06.18564151 +0000 UTC m=+23.218820417"
Sep 18 21:51:06.220525 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.220436    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-976mf" podStartSLOduration=11.349156238 podStartE2EDuration="21.220423848s" totalImagesPullingTime="9.87126761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:06.186072349 +0000 UTC m=+23.219251252" watchObservedRunningTime="2026-09-18 21:51:06.220423848 +0000 UTC m=+23.253602742"
Sep 18 21:51:06.263584 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.263536    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2wm8s" podStartSLOduration=4.206429733 podStartE2EDuration="21.263521313s" totalImagesPullingTime="17.05709158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:06.220883458 +0000 UTC m=+23.254062362" watchObservedRunningTime="2026-09-18 21:51:06.263521313 +0000 UTC m=+23.296700205"
Sep 18 21:51:06.329150 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.329107    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r26z5" podStartSLOduration=4.269689492 podStartE2EDuration="21.329092377s" totalImagesPullingTime="17.059402885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:06.328266914 +0000 UTC m=+23.361445807" watchObservedRunningTime="2026-09-18 21:51:06.329092377 +0000 UTC m=+23.362271260"
Sep 18 21:51:06.329285 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.329218    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c4kct" podStartSLOduration=4.272692346 podStartE2EDuration="21.329211773s" totalImagesPullingTime="17.056519427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:06.263991611 +0000 UTC m=+23.297170506" watchObservedRunningTime="2026-09-18 21:51:06.329211773 +0000 UTC m=+23.362390701"
Sep 18 21:51:06.914001 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.913966    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:51:06.914587 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.914557    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:51:06.924463 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.924351    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T21:51:06.129801599Z","UUID":"d534b5da-eba7-4eed-8d5c-c57d20ea79c1","Handler":null,"Name":"","Endpoint":""}
Sep 18 21:51:06.927182 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.927159    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 21:51:06.927300 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.927195    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 21:51:06.968808 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:06.968783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:06.968980 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:06.968904    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:07.102365 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:07.102332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt" event={"ID":"0caea700-b5e5-42c9-97ec-66c0c5bf19b8","Type":"ContainerStarted","Data":"02c4d4b0336a21db35b738d170ff98c0cdf4a371ebeaa3ce36fdfb6c5ea912dc"}
Sep 18 21:51:07.104185 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:07.104149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7mx7p" event={"ID":"8b5c6a52-d31b-474f-b791-8e5f9b3254fe","Type":"ContainerStarted","Data":"9092d4191d8cc4504931152eb49f5704731a143f1cfb34a6de9a08a037bd0a21"}
Sep 18 21:51:07.126918 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:07.126856    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7mx7p" podStartSLOduration=5.066239037 podStartE2EDuration="22.126838002s" totalImagesPullingTime="17.060598965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:07.126256427 +0000 UTC m=+24.159435310" watchObservedRunningTime="2026-09-18 21:51:07.126838002 +0000 UTC m=+24.160016896"
Sep 18 21:51:07.968503 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:07.968477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:07.968645 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:07.968608    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:07.968693 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:07.968662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:07.968760 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:07.968743    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:08.110941 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:08.110873    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 21:51:08.111393 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:08.111318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"51e344bab4f307b177a9d213d830a8cfb7f805a2290542a5a053609d29cc7742"}
Sep 18 21:51:08.113627 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:08.113602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt" event={"ID":"0caea700-b5e5-42c9-97ec-66c0c5bf19b8","Type":"ContainerStarted","Data":"f2cc38b77697ecd1324de61956885a31e2de393ac12182d14fb689aa39c6eb78"}
Sep 18 21:51:08.113756 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:08.113637    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 21:51:08.136695 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:08.136653    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zg8wt" podStartSLOduration=4.15364936 podStartE2EDuration="23.136637071s" totalImagesPullingTime="18.982987711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:08.136320301 +0000 UTC m=+25.169499199" watchObservedRunningTime="2026-09-18 21:51:08.136637071 +0000 UTC m=+25.169815964"
Sep 18 21:51:08.968370 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:08.968333    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:08.968562 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:08.968468    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:09.968690 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:09.968656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:09.969175 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:09.968664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:09.969175 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:09.968800    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:09.969175 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:09.968873    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:10.612418 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:10.612228    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:51:10.612564 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:10.612535    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 21:51:10.613323 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:10.613231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-976mf"
Sep 18 21:51:10.968251 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:10.968221    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:10.968398 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:10.968325    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:11.123852 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.123809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 21:51:11.124721 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.124207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"9d6c0e827751388dbde9db37cb10474c175733a434f52358ee79fb706abbfeff"}
Sep 18 21:51:11.124721 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.124462    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:51:11.124721 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.124688    2582 scope.go:117] "RemoveContainer" containerID="a4c854cecf71f0c7582e50e2b62affc4dc826ab0e3b9fc91a9a6e252e74563b0"
Sep 18 21:51:11.126207 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.126181    2582 generic.go:356] "Generic (PLEG): container finished" podID="847de099-1e25-441a-8dff-e4d9378896bd" containerID="0edd1ed8e1d8de9cd415196c6802c81441e98a69e78b80fc0a1f48703422fa7f" exitCode=0
Sep 18 21:51:11.126323 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.126274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerDied","Data":"0edd1ed8e1d8de9cd415196c6802c81441e98a69e78b80fc0a1f48703422fa7f"}
Sep 18 21:51:11.141379 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.141363    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:51:11.968933 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.968896    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:11.969119 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:11.968908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:11.969119 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:11.969076    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:11.969219 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:11.969138    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:12.132646 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.132475    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 21:51:12.133053 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.132985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" event={"ID":"8e53a3c0-3b7d-4d05-9357-c7d53f7d096b","Type":"ContainerStarted","Data":"c9afba73e245fc6ac89e515710e1ca9af3af0f1955d69be1cf3229b07714e9a4"}
Sep 18 21:51:12.133234 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.133216    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:51:12.133304 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.133286    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:51:12.149016 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.148991    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:51:12.213705 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.213659    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f" podStartSLOduration=10.128974014 podStartE2EDuration="27.213640046s" totalImagesPullingTime="17.084666032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:12.170289201 +0000 UTC m=+29.203468096" watchObservedRunningTime="2026-09-18 21:51:12.213640046 +0000 UTC m=+29.246818938"
Sep 18 21:51:12.504612 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.504581    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwtqb"]
Sep 18 21:51:12.504756 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.504708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:12.504836 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:12.504813    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:12.507153 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.507130    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x84ph"]
Sep 18 21:51:12.507234 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.507209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:12.507307 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:12.507284    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:12.507901 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.507879    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gddqh"]
Sep 18 21:51:12.508002 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:12.507990    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:12.508118 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:12.508096    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:13.137743 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:13.137703    2582 generic.go:356] "Generic (PLEG): container finished" podID="847de099-1e25-441a-8dff-e4d9378896bd" containerID="a04720934c1ceb3a1e70cf05e6e542d3a2188a88f849789d6939c77bd90b04dd" exitCode=0
Sep 18 21:51:13.138134 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:13.137781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerDied","Data":"a04720934c1ceb3a1e70cf05e6e542d3a2188a88f849789d6939c77bd90b04dd"}
Sep 18 21:51:13.968378 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:13.968349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:13.968520 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:13.968478    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:13.968561 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:13.968534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:13.968641 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:13.968623    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:14.969464 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:14.969432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:14.969792 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:14.969514    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:15.148912 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:15.148877    2582 generic.go:356] "Generic (PLEG): container finished" podID="847de099-1e25-441a-8dff-e4d9378896bd" containerID="1d93eafff7a467a1b7bd8edfe2bfe7d158743ae64bd905cd8eb6130a9d5befdf" exitCode=0
Sep 18 21:51:15.149094 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:15.148923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerDied","Data":"1d93eafff7a467a1b7bd8edfe2bfe7d158743ae64bd905cd8eb6130a9d5befdf"}
Sep 18 21:51:15.968315 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:15.968280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:15.968498 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:15.968336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:15.968498 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:15.968402    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:15.968635 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:15.968512    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:16.968379 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:16.968185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:16.968780 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:16.968489    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gddqh" podUID="299a1b6f-acaa-4fe6-a60f-0ef9f999970f"
Sep 18 21:51:17.968089 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:17.968004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:17.968089 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:17.968028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:17.968288 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:17.968156    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwtqb" podUID="52d1076e-4c86-496e-a462-154feab9b8b9"
Sep 18 21:51:17.968335 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:17.968283    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x84ph" podUID="d421ac05-a1cc-4336-b61b-bf5228b0573d"
Sep 18 21:51:18.021149 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.021117    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-209.ec2.internal" event="NodeReady"
Sep 18 21:51:18.021590 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.021268    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 21:51:18.081193 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.081163    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c4tmv"]
Sep 18 21:51:18.105492 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.105457    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wjwjl"]
Sep 18 21:51:18.105656 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.105594    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.108640 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.108370    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 21:51:18.108640 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.108572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gb4w8\""
Sep 18 21:51:18.110202 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.109849    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 21:51:18.122174 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.122148    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4tmv"]
Sep 18 21:51:18.122262 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.122191    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjwjl"]
Sep 18 21:51:18.122262 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.122254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.127473 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.127439    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 21:51:18.127473 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.127439    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 21:51:18.127646 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.127440    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 21:51:18.127646 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.127534    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vc2r9\""
Sep 18 21:51:18.187269 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.187236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.187458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.187291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc847e3d-5072-4c7d-9ae0-f8fc928ab82f-cert\") pod \"ingress-canary-wjwjl\" (UID: \"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f\") " pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.187458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.187320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-kube-api-access-2xnlq\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.187458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.187349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8pwr\" (UniqueName: \"kubernetes.io/projected/bc847e3d-5072-4c7d-9ae0-f8fc928ab82f-kube-api-access-q8pwr\") pod \"ingress-canary-wjwjl\" (UID: \"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f\") " pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.187458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.187426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-config-volume\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.187458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.187451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-tmp-dir\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.288716 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.288632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-config-volume\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.288716 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.288672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-tmp-dir\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.288716 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.288712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.289000 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.288743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc847e3d-5072-4c7d-9ae0-f8fc928ab82f-cert\") pod \"ingress-canary-wjwjl\" (UID: \"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f\") " pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.289000 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.288764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-kube-api-access-2xnlq\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.289000 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.288789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8pwr\" (UniqueName: \"kubernetes.io/projected/bc847e3d-5072-4c7d-9ae0-f8fc928ab82f-kube-api-access-q8pwr\") pod \"ingress-canary-wjwjl\" (UID: \"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f\") " pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.289156 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.289010    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:18.289156 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.289086    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:51:18.789063381 +0000 UTC m=+35.822242260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:18.289156 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.289138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-tmp-dir\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.289384 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.289360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-config-volume\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.293536 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.293513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc847e3d-5072-4c7d-9ae0-f8fc928ab82f-cert\") pod \"ingress-canary-wjwjl\" (UID: \"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f\") " pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.299476 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.299453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8pwr\" (UniqueName: \"kubernetes.io/projected/bc847e3d-5072-4c7d-9ae0-f8fc928ab82f-kube-api-access-q8pwr\") pod \"ingress-canary-wjwjl\" (UID: \"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f\") " pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.299637 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.299618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xnlq\" (UniqueName: \"kubernetes.io/projected/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-kube-api-access-2xnlq\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.431218 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.431177    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjwjl"
Sep 18 21:51:18.592293 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.592189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:18.592293 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.592242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:18.592508 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.592355    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:18.592508 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.592355    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:18.592508 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.592465    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret podName:299a1b6f-acaa-4fe6-a60f-0ef9f999970f nodeName:}" failed. No retries permitted until 2026-09-18 21:51:50.592443567 +0000 UTC m=+67.625622448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret") pod "global-pull-secret-syncer-gddqh" (UID: "299a1b6f-acaa-4fe6-a60f-0ef9f999970f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 21:51:18.592638 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.592546    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:51:50.592502543 +0000 UTC m=+67.625681427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 21:51:18.693551 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.693511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:18.693716 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.693696    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 21:51:18.693761 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.693723    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 21:51:18.693761 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.693737    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lrpfp for pod openshift-network-diagnostics/network-check-target-x84ph: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:18.693841 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.693792    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp podName:d421ac05-a1cc-4336-b61b-bf5228b0573d nodeName:}" failed. No retries permitted until 2026-09-18 21:51:50.693776787 +0000 UTC m=+67.726955660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lrpfp" (UniqueName: "kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp") pod "network-check-target-x84ph" (UID: "d421ac05-a1cc-4336-b61b-bf5228b0573d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 21:51:18.794425 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.794388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:18.794595 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.794554    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:18.794652 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:18.794611    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:51:19.794595756 +0000 UTC m=+36.827774632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:18.968487 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.968449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:18.971440 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:18.971413    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:51:19.378816 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.378787    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjwjl"]
Sep 18 21:51:19.383204 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:51:19.383176    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc847e3d_5072_4c7d_9ae0_f8fc928ab82f.slice/crio-4131935a9556bc413d4eb03a2ea123f149ea6f4f831a83ae1758ee19f6740c52 WatchSource:0}: Error finding container 4131935a9556bc413d4eb03a2ea123f149ea6f4f831a83ae1758ee19f6740c52: Status 404 returned error can't find the container with id 4131935a9556bc413d4eb03a2ea123f149ea6f4f831a83ae1758ee19f6740c52
Sep 18 21:51:19.802674 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.802433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:19.802880 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:19.802621    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:19.802880 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:19.802808    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:51:21.802782523 +0000 UTC m=+38.835961417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:19.968573 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.968468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:19.968820 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.968792    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:19.974490 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.974464    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:51:19.974639 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.974509    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:51:19.975091 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.975073    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgp2x\""
Sep 18 21:51:19.975285 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.975269    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-46k8l\""
Sep 18 21:51:19.975462 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:19.975442    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:51:20.166149 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:20.166120    2582 generic.go:356] "Generic (PLEG): container finished" podID="847de099-1e25-441a-8dff-e4d9378896bd" containerID="52e42cf3bd704bf8ee187bc609c587ecf2abd09ce39ce11316fd926156451c03" exitCode=0
Sep 18 21:51:20.166453 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:20.166211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerDied","Data":"52e42cf3bd704bf8ee187bc609c587ecf2abd09ce39ce11316fd926156451c03"}
Sep 18 21:51:20.167695 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:20.167671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjwjl" event={"ID":"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f","Type":"ContainerStarted","Data":"4131935a9556bc413d4eb03a2ea123f149ea6f4f831a83ae1758ee19f6740c52"}
Sep 18 21:51:21.173956 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:21.173894    2582 generic.go:356] "Generic (PLEG): container finished" podID="847de099-1e25-441a-8dff-e4d9378896bd" containerID="fdccdc6072f274e2524f81056839a569d212bb5fac65738b87a41fe65f5c3b72" exitCode=0
Sep 18 21:51:21.174503 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:21.173966    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerDied","Data":"fdccdc6072f274e2524f81056839a569d212bb5fac65738b87a41fe65f5c3b72"}
Sep 18 21:51:21.822147 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:21.822111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:21.822315 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:21.822285    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:21.822376 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:21.822364    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:51:25.822342207 +0000 UTC m=+42.855521102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:22.179369 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:22.179335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" event={"ID":"847de099-1e25-441a-8dff-e4d9378896bd","Type":"ContainerStarted","Data":"40881509750a93c792df0e9c437f0247ccf60f2866f7ef84c576a376dae7bc9e"}
Sep 18 21:51:22.180781 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:22.180756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjwjl" event={"ID":"bc847e3d-5072-4c7d-9ae0-f8fc928ab82f","Type":"ContainerStarted","Data":"1e4e2682ace55223e90a334eefb6b0a0f7a588a63dd4232f2c889fcb9474f147"}
Sep 18 21:51:22.216863 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:22.216819    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fn6vl" podStartSLOduration=9.099183108 podStartE2EDuration="37.216803503s" totalImagesPullingTime="28.117620395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:22.216544881 +0000 UTC m=+39.249723774" watchObservedRunningTime="2026-09-18 21:51:22.216803503 +0000 UTC m=+39.249982397"
Sep 18 21:51:22.253960 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:22.253887    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wjwjl" podStartSLOduration=1.926851472 podStartE2EDuration="4.253871998s" totalImagesPullingTime="2.327020526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:22.252635759 +0000 UTC m=+39.285814655" watchObservedRunningTime="2026-09-18 21:51:22.253871998 +0000 UTC m=+39.287050892"
Sep 18 21:51:25.852438 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:25.852401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:25.852803 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:25.852522    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:25.852803 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:25.852573    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:51:33.852560485 +0000 UTC m=+50.885739356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:33.909577 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:33.909539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:33.909972 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:33.909688    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:33.909972 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:33.909766    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:51:49.909749202 +0000 UTC m=+66.942928074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:37.010496 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.010460    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"]
Sep 18 21:51:37.032781 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.032755    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"]
Sep 18 21:51:37.032912 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.032859    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.036621 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.036593    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 21:51:37.037314 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.037295    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 21:51:37.037475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.037374    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5rdx4\""
Sep 18 21:51:37.037475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.037374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 21:51:37.037475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.037434    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 21:51:37.132996 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.132957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ml9t\" (UniqueName: \"kubernetes.io/projected/1d9f8356-7353-4a98-b76c-7ce05a3b3d5c-kube-api-access-6ml9t\") pod \"managed-serviceaccount-addon-agent-74b79f844-4dhv6\" (UID: \"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.133208 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.133047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d9f8356-7353-4a98-b76c-7ce05a3b3d5c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b79f844-4dhv6\" (UID: \"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.233835 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.233801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ml9t\" (UniqueName: \"kubernetes.io/projected/1d9f8356-7353-4a98-b76c-7ce05a3b3d5c-kube-api-access-6ml9t\") pod \"managed-serviceaccount-addon-agent-74b79f844-4dhv6\" (UID: \"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.234018 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.233889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d9f8356-7353-4a98-b76c-7ce05a3b3d5c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b79f844-4dhv6\" (UID: \"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.237534 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.237501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1d9f8356-7353-4a98-b76c-7ce05a3b3d5c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-74b79f844-4dhv6\" (UID: \"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.242825 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.242803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ml9t\" (UniqueName: \"kubernetes.io/projected/1d9f8356-7353-4a98-b76c-7ce05a3b3d5c-kube-api-access-6ml9t\") pod \"managed-serviceaccount-addon-agent-74b79f844-4dhv6\" (UID: \"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.350476 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.350446    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"
Sep 18 21:51:37.652270 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:37.652097    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6"]
Sep 18 21:51:38.219563 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:38.219526    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6" event={"ID":"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c","Type":"ContainerStarted","Data":"a9fb120ef2b2928375d25cf0f6b945d7206f0a739da7fea73e2af6e74ef781e5"}
Sep 18 21:51:41.228498 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:41.228460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6" event={"ID":"1d9f8356-7353-4a98-b76c-7ce05a3b3d5c","Type":"ContainerStarted","Data":"8e4658059095c568da541dc814a184c78ec5264cf4a18ea619690ff5e1a5aec4"}
Sep 18 21:51:41.248141 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:41.248093    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-74b79f844-4dhv6" podStartSLOduration=2.261867728 podStartE2EDuration="5.248076808s" totalImagesPullingTime="2.98620908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:41.247011884 +0000 UTC m=+58.280190779" watchObservedRunningTime="2026-09-18 21:51:41.248076808 +0000 UTC m=+58.281255702"
Sep 18 21:51:44.156376 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:44.156342    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rmj6f"
Sep 18 21:51:49.924719 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:49.924664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:51:49.925133 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:49.924821    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:51:49.925133 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:49.924897    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:21.92487791 +0000 UTC m=+98.958056797 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:51:50.628412 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.628371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:50.628561 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.628422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:51:50.633199 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.633175    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 21:51:50.633304 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.633288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 21:51:50.638997 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:50.638981    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 21:51:50.639048 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:51:50.639037    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs podName:52d1076e-4c86-496e-a462-154feab9b8b9 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:54.639021669 +0000 UTC m=+131.672200546 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs") pod "network-metrics-daemon-nwtqb" (UID: "52d1076e-4c86-496e-a462-154feab9b8b9") : secret "metrics-daemon-secret" not found
Sep 18 21:51:50.640967 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.640929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a1b6f-acaa-4fe6-a60f-0ef9f999970f-original-pull-secret\") pod \"global-pull-secret-syncer-gddqh\" (UID: \"299a1b6f-acaa-4fe6-a60f-0ef9f999970f\") " pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:50.729311 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.729271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:50.732146 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.732130    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 21:51:50.743081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.743059    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 21:51:50.753642 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.753615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrpfp\" (UniqueName: \"kubernetes.io/projected/d421ac05-a1cc-4336-b61b-bf5228b0573d-kube-api-access-lrpfp\") pod \"network-check-target-x84ph\" (UID: \"d421ac05-a1cc-4336-b61b-bf5228b0573d\") " pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:50.778546 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.778521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gddqh"
Sep 18 21:51:50.883941 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.883869    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-46k8l\""
Sep 18 21:51:50.892269 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:50.892250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:51.049383 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:51.049351    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gddqh"]
Sep 18 21:51:51.053933 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:51:51.053905    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299a1b6f_acaa_4fe6_a60f_0ef9f999970f.slice/crio-8a14f27bb02a7eff55dd4ba44ec7cf58524cf5b829626dac6b3d52dafb4b5adf WatchSource:0}: Error finding container 8a14f27bb02a7eff55dd4ba44ec7cf58524cf5b829626dac6b3d52dafb4b5adf: Status 404 returned error can't find the container with id 8a14f27bb02a7eff55dd4ba44ec7cf58524cf5b829626dac6b3d52dafb4b5adf
Sep 18 21:51:51.160188 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:51.160104    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x84ph"]
Sep 18 21:51:51.163867 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:51:51.163838    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd421ac05_a1cc_4336_b61b_bf5228b0573d.slice/crio-cc9e81aee99de99cec67af5aa530ff1f5a04f4b24632913bd9332e14ec60a9d0 WatchSource:0}: Error finding container cc9e81aee99de99cec67af5aa530ff1f5a04f4b24632913bd9332e14ec60a9d0: Status 404 returned error can't find the container with id cc9e81aee99de99cec67af5aa530ff1f5a04f4b24632913bd9332e14ec60a9d0
Sep 18 21:51:51.252528 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:51.252491    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x84ph" event={"ID":"d421ac05-a1cc-4336-b61b-bf5228b0573d","Type":"ContainerStarted","Data":"cc9e81aee99de99cec67af5aa530ff1f5a04f4b24632913bd9332e14ec60a9d0"}
Sep 18 21:51:51.253564 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:51.253539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gddqh" event={"ID":"299a1b6f-acaa-4fe6-a60f-0ef9f999970f","Type":"ContainerStarted","Data":"8a14f27bb02a7eff55dd4ba44ec7cf58524cf5b829626dac6b3d52dafb4b5adf"}
Sep 18 21:51:56.270000 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:56.269968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x84ph" event={"ID":"d421ac05-a1cc-4336-b61b-bf5228b0573d","Type":"ContainerStarted","Data":"9ae2c4a398608619e7fada569d6ca1d5676bd1722a2d2d010db16366823e4bbc"}
Sep 18 21:51:56.270475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:56.270027    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:51:56.271435 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:56.271406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gddqh" event={"ID":"299a1b6f-acaa-4fe6-a60f-0ef9f999970f","Type":"ContainerStarted","Data":"e8985fcf8b28b78983122392ee987feeb0de3dee48b0aa3e418fe1fa0e9b0e09"}
Sep 18 21:51:56.290828 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:51:56.290783    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x84ph" podStartSLOduration=67.059200322 podStartE2EDuration="1m11.290769438s" totalImagesPullingTime="4.231569116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:56.290034428 +0000 UTC m=+73.323213336" watchObservedRunningTime="2026-09-18 21:51:56.290769438 +0000 UTC m=+73.323948332"
Sep 18 21:52:21.393913 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.393781    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gddqh" podStartSLOduration=92.047861557 podStartE2EDuration="1m36.393763906s" totalImagesPullingTime="4.345902349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:51:56.305417425 +0000 UTC m=+73.338596319" watchObservedRunningTime="2026-09-18 21:52:21.393763906 +0000 UTC m=+98.426942778"
Sep 18 21:52:21.394385 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.394210    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tv5mn"]
Sep 18 21:52:21.400395 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.400377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.404805 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.404782    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 21:52:21.404984 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.404967    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 21:52:21.405657 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.405634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmpgg\""
Sep 18 21:52:21.405756 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.405697    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 21:52:21.405756 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.405697    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 21:52:21.415694 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.415671    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tv5mn"]
Sep 18 21:52:21.438226 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.438201    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99nc5\" (UniqueName: \"kubernetes.io/projected/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-api-access-99nc5\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.438330 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.438243    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.438330 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.438276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.438444 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.438393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3838c43-2bb5-4a08-bde1-9844e7401f3a-data-volume\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.438444 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.438432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3838c43-2bb5-4a08-bde1-9844e7401f3a-crio-socket\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.539018 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.538983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3838c43-2bb5-4a08-bde1-9844e7401f3a-data-volume\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.539018 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.539017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3838c43-2bb5-4a08-bde1-9844e7401f3a-crio-socket\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.539225 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.539047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99nc5\" (UniqueName: \"kubernetes.io/projected/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-api-access-99nc5\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.539225 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.539136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3838c43-2bb5-4a08-bde1-9844e7401f3a-crio-socket\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.539225 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.539176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.539382 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.539228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.539382 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:21.539279    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:21.539382 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:21.539296    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:21.539382 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:21.539358    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:22.039337629 +0000 UTC m=+99.072516512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:21.539382 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:21.539373    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:22.0393669 +0000 UTC m=+99.072545776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:21.539382 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.539352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3838c43-2bb5-4a08-bde1-9844e7401f3a-data-volume\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.548473 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.548451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99nc5\" (UniqueName: \"kubernetes.io/projected/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-api-access-99nc5\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:21.942721 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:21.942694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:52:21.942925 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:21.942869    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 21:52:21.943022 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:21.942971    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls podName:9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c nodeName:}" failed. No retries permitted until 2026-09-18 21:53:25.942930834 +0000 UTC m=+162.976109709 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls") pod "dns-default-c4tmv" (UID: "9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c") : secret "dns-default-metrics-tls" not found
Sep 18 21:52:22.043232 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:22.043185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:22.043397 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:22.043255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:22.043397 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:22.043331    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:22.043397 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:22.043350    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:22.043397 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:22.043393    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:23.043375821 +0000 UTC m=+100.076554694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:22.043605 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:22.043408    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:23.043401524 +0000 UTC m=+100.076580396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:23.049729 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:23.049695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:23.049729 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:23.049736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:23.050257 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:23.049833    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:23.050257 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:23.049840    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:23.050257 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:23.049893    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:25.049880227 +0000 UTC m=+102.083059115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:23.050257 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:23.049907    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:25.049901239 +0000 UTC m=+102.083080111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:23.075842 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:23.075817    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wm8s_310ea0c0-e1ce-4eb1-871f-33ef46f48827/dns-node-resolver/0.log"
Sep 18 21:52:23.876640 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:23.876612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4kct_6e2d73b3-52ef-4f05-839a-051bab482a3c/node-ca/0.log"
Sep 18 21:52:24.875993 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:24.875966    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wjwjl_bc847e3d-5072-4c7d-9ae0-f8fc928ab82f/serve-healthcheck-canary/0.log"
Sep 18 21:52:25.064576 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:25.064543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:25.064736 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:25.064589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:25.064736 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:25.064646    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:25.064736 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:25.064703    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.064690171 +0000 UTC m=+106.097869043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:25.064736 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:25.064649    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:25.064873 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:25.064794    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.064779337 +0000 UTC m=+106.097958209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:27.276445 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:27.276417    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x84ph"
Sep 18 21:52:28.171900 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.171844    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"]
Sep 18 21:52:28.200746 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.200717    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"]
Sep 18 21:52:28.200895 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.200842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.203161 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.203132    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 21:52:28.203161 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.203151    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pjjtt\""
Sep 18 21:52:28.203346 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.203169    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:52:28.203346 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.203182    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 21:52:28.203346 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.203187    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 21:52:28.224735 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.224709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"
Sep 18 21:52:28.224735 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.224727    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-q4dqp"]
Sep 18 21:52:28.227175 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.227155    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:52:28.227303 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.227177    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 21:52:28.227303 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.227257    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zv4m4\""
Sep 18 21:52:28.240712 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.240694    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"]
Sep 18 21:52:28.240835 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.240721    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"]
Sep 18 21:52:28.240835 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.240739    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-q4dqp"]
Sep 18 21:52:28.240835 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.240796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.243215 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.243198    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 21:52:28.243369 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.243357    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 21:52:28.243436 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.243359    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:52:28.243436 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.243361    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dcntc\""
Sep 18 21:52:28.243436 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.243416    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 21:52:28.247676 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.247660    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 21:52:28.275300 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.275205    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"]
Sep 18 21:52:28.284075 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67298389-9b58-4a22-8f3d-610f24790eb1-serving-cert\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.284350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f64f8e8-176b-493f-8839-e1d04f98be13-config\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.284350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhl7\" (UniqueName: \"kubernetes.io/projected/6f64f8e8-176b-493f-8839-e1d04f98be13-kube-api-access-nlhl7\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.284350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67298389-9b58-4a22-8f3d-610f24790eb1-trusted-ca\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.284350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f64f8e8-176b-493f-8839-e1d04f98be13-serving-cert\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.284350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmmx7\" (UniqueName: \"kubernetes.io/projected/e73cb2a4-8da4-45ef-8aca-7e784ce00dd4-kube-api-access-mmmx7\") pod \"volume-data-source-validator-5c4db8bdf9-2rkn8\" (UID: \"e73cb2a4-8da4-45ef-8aca-7e784ce00dd4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"
Sep 18 21:52:28.284350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67298389-9b58-4a22-8f3d-610f24790eb1-config\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.284350 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.284276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv55p\" (UniqueName: \"kubernetes.io/projected/67298389-9b58-4a22-8f3d-610f24790eb1-kube-api-access-dv55p\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.296796 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.296774    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"]
Sep 18 21:52:28.296918 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.296894    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.299346 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.299325    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 21:52:28.299346 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.299340    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 21:52:28.299504 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.299475    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:52:28.300010 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.299992    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 21:52:28.300120 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.300101    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6jxkl\""
Sep 18 21:52:28.314999 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.314977    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-657b5f585b-96g68"]
Sep 18 21:52:28.315109 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.315058    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:28.317382 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.317368    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8db9g\""
Sep 18 21:52:28.317752 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.317735    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:52:28.317835 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.317780    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 21:52:28.318007 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.317992    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 21:52:28.328933 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.328916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"]
Sep 18 21:52:28.329038 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.328961    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-657b5f585b-96g68"]
Sep 18 21:52:28.329038 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.328982    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"]
Sep 18 21:52:28.329038 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.329028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.331312 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.331275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 21:52:28.331422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.331367    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 21:52:28.331422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.331404    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 21:52:28.331577 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.331424    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 21:52:28.331577 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.331465    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 21:52:28.331577 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.331500    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 21:52:28.331767 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.331754    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dhhxf\""
Sep 18 21:52:28.384566 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.384682 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f64f8e8-176b-493f-8839-e1d04f98be13-config\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.384682 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f64f8e8-176b-493f-8839-e1d04f98be13-serving-cert\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.384682 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhl7\" (UniqueName: \"kubernetes.io/projected/6f64f8e8-176b-493f-8839-e1d04f98be13-kube-api-access-nlhl7\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.384682 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384673    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e494d22-0a87-4169-ac16-40037cdeaee6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.384869 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.384869 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-default-certificate\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.384869 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67298389-9b58-4a22-8f3d-610f24790eb1-config\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.384869 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmmx7\" (UniqueName: \"kubernetes.io/projected/e73cb2a4-8da4-45ef-8aca-7e784ce00dd4-kube-api-access-mmmx7\") pod \"volume-data-source-validator-5c4db8bdf9-2rkn8\" (UID: \"e73cb2a4-8da4-45ef-8aca-7e784ce00dd4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"
Sep 18 21:52:28.385101 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.384974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67298389-9b58-4a22-8f3d-610f24790eb1-serving-cert\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.385101 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385011    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wf9j\" (UniqueName: \"kubernetes.io/projected/3e494d22-0a87-4169-ac16-40037cdeaee6-kube-api-access-9wf9j\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.385101 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4v2r\" (UniqueName: \"kubernetes.io/projected/3529e86f-3aea-4f0e-9f49-6250abcb042c-kube-api-access-z4v2r\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.385101 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2p6s\" (UniqueName: \"kubernetes.io/projected/d23045ab-140c-402d-a5c7-eea4a144eda8-kube-api-access-c2p6s\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:28.385101 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67298389-9b58-4a22-8f3d-610f24790eb1-trusted-ca\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.385349 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv55p\" (UniqueName: \"kubernetes.io/projected/67298389-9b58-4a22-8f3d-610f24790eb1-kube-api-access-dv55p\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.385349 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-stats-auth\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.385349 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:28.385349 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e494d22-0a87-4169-ac16-40037cdeaee6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.385349 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f64f8e8-176b-493f-8839-e1d04f98be13-config\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.385598 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/67298389-9b58-4a22-8f3d-610f24790eb1-config\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.385789 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.385772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67298389-9b58-4a22-8f3d-610f24790eb1-trusted-ca\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.386961 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.386916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f64f8e8-176b-493f-8839-e1d04f98be13-serving-cert\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.387253 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.387234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/67298389-9b58-4a22-8f3d-610f24790eb1-serving-cert\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.394066 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.394027    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhl7\" (UniqueName: \"kubernetes.io/projected/6f64f8e8-176b-493f-8839-e1d04f98be13-kube-api-access-nlhl7\") pod \"service-ca-operator-57cdc4b768-46bsl\" (UID: \"6f64f8e8-176b-493f-8839-e1d04f98be13\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.394527 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.394203    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv55p\" (UniqueName: \"kubernetes.io/projected/67298389-9b58-4a22-8f3d-610f24790eb1-kube-api-access-dv55p\") pod \"console-operator-fd4bb46c-q4dqp\" (UID: \"67298389-9b58-4a22-8f3d-610f24790eb1\") " pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.394527 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.394390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmmx7\" (UniqueName: \"kubernetes.io/projected/e73cb2a4-8da4-45ef-8aca-7e784ce00dd4-kube-api-access-mmmx7\") pod \"volume-data-source-validator-5c4db8bdf9-2rkn8\" (UID: \"e73cb2a4-8da4-45ef-8aca-7e784ce00dd4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"
Sep 18 21:52:28.486005 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.485908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.486005 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.485973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e494d22-0a87-4169-ac16-40037cdeaee6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.486005 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.485993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.486257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-default-certificate\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.486257 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.486074    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 21:52:28.486257 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.486139    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:28.986122254 +0000 UTC m=+106.019301126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : secret "router-metrics-certs-default" not found
Sep 18 21:52:28.486257 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.486167    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:28.98614997 +0000 UTC m=+106.019328856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : configmap references non-existent config key: service-ca.crt
Sep 18 21:52:28.486257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wf9j\" (UniqueName: \"kubernetes.io/projected/3e494d22-0a87-4169-ac16-40037cdeaee6-kube-api-access-9wf9j\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.486257 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4v2r\" (UniqueName: \"kubernetes.io/projected/3529e86f-3aea-4f0e-9f49-6250abcb042c-kube-api-access-z4v2r\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.486608 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2p6s\" (UniqueName: \"kubernetes.io/projected/d23045ab-140c-402d-a5c7-eea4a144eda8-kube-api-access-c2p6s\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:28.486608 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-stats-auth\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.486608 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:28.486608 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e494d22-0a87-4169-ac16-40037cdeaee6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.486608 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.486431    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:52:28.486608 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.486490    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls podName:d23045ab-140c-402d-a5c7-eea4a144eda8 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:28.9864759 +0000 UTC m=+106.019654775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-c5vhr" (UID: "d23045ab-140c-402d-a5c7-eea4a144eda8") : secret "samples-operator-tls" not found
Sep 18 21:52:28.486940 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.486919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e494d22-0a87-4169-ac16-40037cdeaee6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.488308 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.488289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e494d22-0a87-4169-ac16-40037cdeaee6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.488531 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.488511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-stats-auth\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.488612 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.488588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-default-certificate\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.500143 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.500123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wf9j\" (UniqueName: \"kubernetes.io/projected/3e494d22-0a87-4169-ac16-40037cdeaee6-kube-api-access-9wf9j\") pod \"kube-storage-version-migrator-operator-55889f7b47-vl67p\" (UID: \"3e494d22-0a87-4169-ac16-40037cdeaee6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.501846 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.501829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2p6s\" (UniqueName: \"kubernetes.io/projected/d23045ab-140c-402d-a5c7-eea4a144eda8-kube-api-access-c2p6s\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:28.506068 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.506046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4v2r\" (UniqueName: \"kubernetes.io/projected/3529e86f-3aea-4f0e-9f49-6250abcb042c-kube-api-access-z4v2r\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.509860 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.509846    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"
Sep 18 21:52:28.532919 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.532898    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"
Sep 18 21:52:28.549206 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.549185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:28.606502 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.606473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"
Sep 18 21:52:28.803532 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.803508    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl"]
Sep 18 21:52:28.805999 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:52:28.805969    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f64f8e8_176b_493f_8839_e1d04f98be13.slice/crio-0fbe8e761de0daaa9be1889c3f7b2d7390b0db6b80a14eafbab6f3bf421f24c1 WatchSource:0}: Error finding container 0fbe8e761de0daaa9be1889c3f7b2d7390b0db6b80a14eafbab6f3bf421f24c1: Status 404 returned error can't find the container with id 0fbe8e761de0daaa9be1889c3f7b2d7390b0db6b80a14eafbab6f3bf421f24c1
Sep 18 21:52:28.847579 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.847546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8"]
Sep 18 21:52:28.850783 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:52:28.850761    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73cb2a4_8da4_45ef_8aca_7e784ce00dd4.slice/crio-54bbfb5958cf9424fd38efcf9d2f4068f019537ca73d3531f8037ed670fe08f3 WatchSource:0}: Error finding container 54bbfb5958cf9424fd38efcf9d2f4068f019537ca73d3531f8037ed670fe08f3: Status 404 returned error can't find the container with id 54bbfb5958cf9424fd38efcf9d2f4068f019537ca73d3531f8037ed670fe08f3
Sep 18 21:52:28.868373 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.868351    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-q4dqp"]
Sep 18 21:52:28.870797 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:52:28.870775    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67298389_9b58_4a22_8f3d_610f24790eb1.slice/crio-3f1ebec01f192e94ecac912d70115cc58d415f16beea42496b79bf1aff1ecc93 WatchSource:0}: Error finding container 3f1ebec01f192e94ecac912d70115cc58d415f16beea42496b79bf1aff1ecc93: Status 404 returned error can't find the container with id 3f1ebec01f192e94ecac912d70115cc58d415f16beea42496b79bf1aff1ecc93
Sep 18 21:52:28.908906 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.908793    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p"]
Sep 18 21:52:28.911099 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:52:28.911077    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e494d22_0a87_4169_ac16_40037cdeaee6.slice/crio-e958b659b8b1606fa19899d3ec5974c59262dc826fa6f6738ea385a946a2f1cc WatchSource:0}: Error finding container e958b659b8b1606fa19899d3ec5974c59262dc826fa6f6738ea385a946a2f1cc: Status 404 returned error can't find the container with id e958b659b8b1606fa19899d3ec5974c59262dc826fa6f6738ea385a946a2f1cc
Sep 18 21:52:28.992200 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.992172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:28.992328 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.992208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.992328 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:28.992233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:28.992328 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.992317    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 21:52:28.992453 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.992323    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.992308798 +0000 UTC m=+107.025487670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : configmap references non-existent config key: service-ca.crt
Sep 18 21:52:28.992453 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.992365    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:52:28.992453 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.992377    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.992364134 +0000 UTC m=+107.025543009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : secret "router-metrics-certs-default" not found
Sep 18 21:52:28.992453 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:28.992405    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls podName:d23045ab-140c-402d-a5c7-eea4a144eda8 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:29.992394112 +0000 UTC m=+107.025573006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-c5vhr" (UID: "d23045ab-140c-402d-a5c7-eea4a144eda8") : secret "samples-operator-tls" not found
Sep 18 21:52:29.093799 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:29.093729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:29.093799 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:29.093765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:29.093979 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:29.093837    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 21:52:29.093979 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:29.093873    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:29.093979 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:29.093879    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:37.093865846 +0000 UTC m=+114.127044719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : configmap "kube-rbac-proxy" not found
Sep 18 21:52:29.093979 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:29.093936    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:37.093918883 +0000 UTC m=+114.127097754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:29.353534 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:29.353446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl" event={"ID":"6f64f8e8-176b-493f-8839-e1d04f98be13","Type":"ContainerStarted","Data":"0fbe8e761de0daaa9be1889c3f7b2d7390b0db6b80a14eafbab6f3bf421f24c1"}
Sep 18 21:52:29.354562 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:29.354536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" event={"ID":"67298389-9b58-4a22-8f3d-610f24790eb1","Type":"ContainerStarted","Data":"3f1ebec01f192e94ecac912d70115cc58d415f16beea42496b79bf1aff1ecc93"}
Sep 18 21:52:29.355651 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:29.355624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8" event={"ID":"e73cb2a4-8da4-45ef-8aca-7e784ce00dd4","Type":"ContainerStarted","Data":"54bbfb5958cf9424fd38efcf9d2f4068f019537ca73d3531f8037ed670fe08f3"}
Sep 18 21:52:29.356643 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:29.356624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p" event={"ID":"3e494d22-0a87-4169-ac16-40037cdeaee6","Type":"ContainerStarted","Data":"e958b659b8b1606fa19899d3ec5974c59262dc826fa6f6738ea385a946a2f1cc"}
Sep 18 21:52:30.003580 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:30.003543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:30.003771 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:30.003598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:30.003771 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:30.003635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:30.003771 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:30.003752    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:52:30.003966 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:30.003798    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:32.003777438 +0000 UTC m=+109.036956309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : configmap references non-existent config key: service-ca.crt
Sep 18 21:52:30.003966 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:30.003879    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls podName:d23045ab-140c-402d-a5c7-eea4a144eda8 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:32.003854659 +0000 UTC m=+109.037033536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-c5vhr" (UID: "d23045ab-140c-402d-a5c7-eea4a144eda8") : secret "samples-operator-tls" not found
Sep 18 21:52:30.003966 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:30.003888    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 21:52:30.003966 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:30.003930    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:32.003917886 +0000 UTC m=+109.037096772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : secret "router-metrics-certs-default" not found
Sep 18 21:52:30.364461 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:30.364396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8" event={"ID":"e73cb2a4-8da4-45ef-8aca-7e784ce00dd4","Type":"ContainerStarted","Data":"0cc5dab9f5c37d200a45daf847cc5661940f1a14d4afb902fca7a874a372083d"}
Sep 18 21:52:32.021678 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.021634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:32.022106 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.021772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:32.022106 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.021807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:32.022106 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:32.021842    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:36.021817916 +0000 UTC m=+113.054996792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : configmap references non-existent config key: service-ca.crt
Sep 18 21:52:32.022106 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:32.021895    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 21:52:32.022106 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:32.021960    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:36.02193057 +0000 UTC m=+113.055109457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : secret "router-metrics-certs-default" not found
Sep 18 21:52:32.022106 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:32.022016    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:52:32.022106 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:32.022048    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls podName:d23045ab-140c-402d-a5c7-eea4a144eda8 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:36.022037471 +0000 UTC m=+113.055216366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-c5vhr" (UID: "d23045ab-140c-402d-a5c7-eea4a144eda8") : secret "samples-operator-tls" not found
Sep 18 21:52:32.372495 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.372457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p" event={"ID":"3e494d22-0a87-4169-ac16-40037cdeaee6","Type":"ContainerStarted","Data":"11d34dfcdfe4666fbd601889e10d8326535cf307fa8efeecd3531e9b85dc4267"}
Sep 18 21:52:32.374058 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.374030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl" event={"ID":"6f64f8e8-176b-493f-8839-e1d04f98be13","Type":"ContainerStarted","Data":"4eb7cf157947c24f71671bb9e797ad7e9e87bcaf3fd59b9a63640de3c9f67646"}
Sep 18 21:52:32.394107 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.394066    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-2rkn8" podStartSLOduration=2.957213743 podStartE2EDuration="4.394051573s" totalImagesPullingTime="1.43683783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:31.398194778 +0000 UTC m=+108.431373675" watchObservedRunningTime="2026-09-18 21:52:32.394051573 +0000 UTC m=+109.427230469"
Sep 18 21:52:32.394213 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.394140    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p" podStartSLOduration=1.926747528 podStartE2EDuration="4.394133881s" totalImagesPullingTime="2.467386353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:32.393443876 +0000 UTC m=+109.426622771" watchObservedRunningTime="2026-09-18 21:52:32.394133881 +0000 UTC m=+109.427312774"
Sep 18 21:52:32.409321 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:32.409271    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl" podStartSLOduration=1.839400556 podStartE2EDuration="4.409251941s" totalImagesPullingTime="2.569851385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:32.408607789 +0000 UTC m=+109.441786683" watchObservedRunningTime="2026-09-18 21:52:32.409251941 +0000 UTC m=+109.442430837"
Sep 18 21:52:33.378393 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:33.378362    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/0.log"
Sep 18 21:52:33.378787 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:33.378408    2582 generic.go:356] "Generic (PLEG): container finished" podID="67298389-9b58-4a22-8f3d-610f24790eb1" containerID="852e8292874dbb6919e14bbb614592d8953b427b98baf584e915758546b38e8c" exitCode=255
Sep 18 21:52:33.378787 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:33.378490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" event={"ID":"67298389-9b58-4a22-8f3d-610f24790eb1","Type":"ContainerDied","Data":"852e8292874dbb6919e14bbb614592d8953b427b98baf584e915758546b38e8c"}
Sep 18 21:52:33.378787 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:33.378687    2582 scope.go:117] "RemoveContainer" containerID="852e8292874dbb6919e14bbb614592d8953b427b98baf584e915758546b38e8c"
Sep 18 21:52:34.383554 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:34.383524    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/1.log"
Sep 18 21:52:34.384000 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:34.383940    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/0.log"
Sep 18 21:52:34.384074 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:34.384002    2582 generic.go:356] "Generic (PLEG): container finished" podID="67298389-9b58-4a22-8f3d-610f24790eb1" containerID="3eafce3fd38da0633e26255857718d3c4039f9446d96bf770cbabe52a040dfc0" exitCode=255
Sep 18 21:52:34.384074 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:34.384047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" event={"ID":"67298389-9b58-4a22-8f3d-610f24790eb1","Type":"ContainerDied","Data":"3eafce3fd38da0633e26255857718d3c4039f9446d96bf770cbabe52a040dfc0"}
Sep 18 21:52:34.384173 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:34.384105    2582 scope.go:117] "RemoveContainer" containerID="852e8292874dbb6919e14bbb614592d8953b427b98baf584e915758546b38e8c"
Sep 18 21:52:34.384284 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:34.384269    2582 scope.go:117] "RemoveContainer" containerID="3eafce3fd38da0633e26255857718d3c4039f9446d96bf770cbabe52a040dfc0"
Sep 18 21:52:34.384458 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:34.384439    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-q4dqp_openshift-console-operator(67298389-9b58-4a22-8f3d-610f24790eb1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" podUID="67298389-9b58-4a22-8f3d-610f24790eb1"
Sep 18 21:52:35.388540 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:35.388509    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/1.log"
Sep 18 21:52:35.389039 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:35.388894    2582 scope.go:117] "RemoveContainer" containerID="3eafce3fd38da0633e26255857718d3c4039f9446d96bf770cbabe52a040dfc0"
Sep 18 21:52:35.389139 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:35.389115    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-q4dqp_openshift-console-operator(67298389-9b58-4a22-8f3d-610f24790eb1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" podUID="67298389-9b58-4a22-8f3d-610f24790eb1"
Sep 18 21:52:36.055201 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:36.055168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:36.055363 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:36.055265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:36.055363 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:36.055287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:36.055473 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:36.055400    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:44.055386142 +0000 UTC m=+121.088565014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : configmap references non-existent config key: service-ca.crt
Sep 18 21:52:36.055473 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:36.055400    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 21:52:36.055473 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:36.055454    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 21:52:36.055473 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:36.055468    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls podName:d23045ab-140c-402d-a5c7-eea4a144eda8 nodeName:}" failed. No retries permitted until 2026-09-18 21:52:44.055450537 +0000 UTC m=+121.088629412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-c5vhr" (UID: "d23045ab-140c-402d-a5c7-eea4a144eda8") : secret "samples-operator-tls" not found
Sep 18 21:52:36.055633 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:36.055521    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:52:44.055503997 +0000 UTC m=+121.088682874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : secret "router-metrics-certs-default" not found
Sep 18 21:52:37.165296 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:37.165248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:37.165296 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:37.165309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:37.165806 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:37.165411    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:37.165806 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:37.165513    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls podName:e3838c43-2bb5-4a08-bde1-9844e7401f3a nodeName:}" failed. No retries permitted until 2026-09-18 21:52:53.16549271 +0000 UTC m=+130.198671596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tv5mn" (UID: "e3838c43-2bb5-4a08-bde1-9844e7401f3a") : secret "insights-runtime-extractor-tls" not found
Sep 18 21:52:37.165806 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:37.165756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3838c43-2bb5-4a08-bde1-9844e7401f3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:38.549320 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:38.549289    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:38.549684 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:38.549348    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:38.549684 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:38.549641    2582 scope.go:117] "RemoveContainer" containerID="3eafce3fd38da0633e26255857718d3c4039f9446d96bf770cbabe52a040dfc0"
Sep 18 21:52:38.549804 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:38.549785    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-q4dqp_openshift-console-operator(67298389-9b58-4a22-8f3d-610f24790eb1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" podUID="67298389-9b58-4a22-8f3d-610f24790eb1"
Sep 18 21:52:44.120209 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:44.120175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:44.120603 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:44.120228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:44.120603 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:44.120254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:44.120603 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:44.120363    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle podName:3529e86f-3aea-4f0e-9f49-6250abcb042c nodeName:}" failed. No retries permitted until 2026-09-18 21:53:00.120343672 +0000 UTC m=+137.153522549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle") pod "router-default-657b5f585b-96g68" (UID: "3529e86f-3aea-4f0e-9f49-6250abcb042c") : configmap references non-existent config key: service-ca.crt
Sep 18 21:52:44.122585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:44.122554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d23045ab-140c-402d-a5c7-eea4a144eda8-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-c5vhr\" (UID: \"d23045ab-140c-402d-a5c7-eea4a144eda8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:44.122692 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:44.122635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3529e86f-3aea-4f0e-9f49-6250abcb042c-metrics-certs\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:52:44.222729 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:44.222701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"
Sep 18 21:52:44.499405 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:44.499329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr"]
Sep 18 21:52:45.419580 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:45.419511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr" event={"ID":"d23045ab-140c-402d-a5c7-eea4a144eda8","Type":"ContainerStarted","Data":"ce3ae3efe7f7a53d4af91c17c4b60f7e2362adbbb93b53accd48135fadfe6dc6"}
Sep 18 21:52:46.424220 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:46.424190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr" event={"ID":"d23045ab-140c-402d-a5c7-eea4a144eda8","Type":"ContainerStarted","Data":"153e08cb73a4b673ad0e5ebb39c270f6fc1f9a2c0db4cf4f5fbb386006ac6532"}
Sep 18 21:52:46.424491 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:46.424251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr" event={"ID":"d23045ab-140c-402d-a5c7-eea4a144eda8","Type":"ContainerStarted","Data":"55b028a4afcb1358a02d69ea6277f63d514919b129f916069dea25d08faa72fc"}
Sep 18 21:52:49.968800 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:49.968766    2582 scope.go:117] "RemoveContainer" containerID="3eafce3fd38da0633e26255857718d3c4039f9446d96bf770cbabe52a040dfc0"
Sep 18 21:52:50.437716 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:50.437687    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 21:52:50.438132 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:50.438117    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/1.log"
Sep 18 21:52:50.438200 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:50.438150    2582 generic.go:356] "Generic (PLEG): container finished" podID="67298389-9b58-4a22-8f3d-610f24790eb1" containerID="0c48c2e8becc3a62ae453647682f760f173d1f4bfd7430bdf097f3644b9a14fb" exitCode=255
Sep 18 21:52:50.438238 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:50.438204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" event={"ID":"67298389-9b58-4a22-8f3d-610f24790eb1","Type":"ContainerDied","Data":"0c48c2e8becc3a62ae453647682f760f173d1f4bfd7430bdf097f3644b9a14fb"}
Sep 18 21:52:50.438285 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:50.438251    2582 scope.go:117] "RemoveContainer" containerID="3eafce3fd38da0633e26255857718d3c4039f9446d96bf770cbabe52a040dfc0"
Sep 18 21:52:50.438534 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:50.438511    2582 scope.go:117] "RemoveContainer" containerID="0c48c2e8becc3a62ae453647682f760f173d1f4bfd7430bdf097f3644b9a14fb"
Sep 18 21:52:50.438698 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:50.438680    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-q4dqp_openshift-console-operator(67298389-9b58-4a22-8f3d-610f24790eb1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" podUID="67298389-9b58-4a22-8f3d-610f24790eb1"
Sep 18 21:52:50.456389 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:50.456348    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-c5vhr" podStartSLOduration=20.689395856 podStartE2EDuration="22.456334402s" totalImagesPullingTime="1.766938546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:47.451321876 +0000 UTC m=+124.484500774" watchObservedRunningTime="2026-09-18 21:52:50.456334402 +0000 UTC m=+127.489513337"
Sep 18 21:52:51.444533 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:51.444501    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 21:52:53.205237 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:53.205192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:53.207402 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:53.207383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3838c43-2bb5-4a08-bde1-9844e7401f3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tv5mn\" (UID: \"e3838c43-2bb5-4a08-bde1-9844e7401f3a\") " pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:53.212093 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:53.212074    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmpgg\""
Sep 18 21:52:53.219972 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:53.219924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tv5mn"
Sep 18 21:52:53.497065 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:53.496992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tv5mn"]
Sep 18 21:52:53.500258 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:52:53.500231    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3838c43_2bb5_4a08_bde1_9844e7401f3a.slice/crio-b6c19de3e54362cc5044c528d35f97c3dcb562bfc7d09d718a2cf12da7f6e4bd WatchSource:0}: Error finding container b6c19de3e54362cc5044c528d35f97c3dcb562bfc7d09d718a2cf12da7f6e4bd: Status 404 returned error can't find the container with id b6c19de3e54362cc5044c528d35f97c3dcb562bfc7d09d718a2cf12da7f6e4bd
Sep 18 21:52:54.457054 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:54.456979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tv5mn" event={"ID":"e3838c43-2bb5-4a08-bde1-9844e7401f3a","Type":"ContainerStarted","Data":"a3e36af88b87fe175f823fc27ef436e4bf6cb40a29d99a13c5eb5657adba7bbb"}
Sep 18 21:52:54.457385 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:54.457057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tv5mn" event={"ID":"e3838c43-2bb5-4a08-bde1-9844e7401f3a","Type":"ContainerStarted","Data":"3b2c65bc9b98f5d589aec18adc0269fb6b7c472e0a6d3c15d524e1f94e7c5583"}
Sep 18 21:52:54.457385 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:54.457078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tv5mn" event={"ID":"e3838c43-2bb5-4a08-bde1-9844e7401f3a","Type":"ContainerStarted","Data":"b6c19de3e54362cc5044c528d35f97c3dcb562bfc7d09d718a2cf12da7f6e4bd"}
Sep 18 21:52:54.718372 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:54.718279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:52:54.721081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:54.721051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52d1076e-4c86-496e-a462-154feab9b8b9-metrics-certs\") pod \"network-metrics-daemon-nwtqb\" (UID: \"52d1076e-4c86-496e-a462-154feab9b8b9\") " pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:52:54.790600 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:54.790566    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgp2x\""
Sep 18 21:52:54.797880 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:54.797856    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwtqb"
Sep 18 21:52:55.095513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.095444    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwtqb"]
Sep 18 21:52:55.231763 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.231733    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5655cdf544-z7pmn"]
Sep 18 21:52:55.235938 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.235918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.238323 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.238302    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 21:52:55.240704 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.240639    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 21:52:55.240932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.240912    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 21:52:55.246680 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.246641    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 21:52:55.253129 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.253107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xmfvd\""
Sep 18 21:52:55.261891 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.261865    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5655cdf544-z7pmn"]
Sep 18 21:52:55.287064 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.287039    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5655cdf544-z7pmn"]
Sep 18 21:52:55.287291 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:55.287260    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-j8jhv registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-j8jhv registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-5655cdf544-z7pmn" podUID="fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"
Sep 18 21:52:55.394233 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:52:55.394202    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d1076e_4c86_496e_a462_154feab9b8b9.slice/crio-92c8598b9da8e3e4c5d1c5d738d5541a2d2233db66553d47d9df25233f431d20 WatchSource:0}: Error finding container 92c8598b9da8e3e4c5d1c5d738d5541a2d2233db66553d47d9df25233f431d20: Status 404 returned error can't find the container with id 92c8598b9da8e3e4c5d1c5d738d5541a2d2233db66553d47d9df25233f431d20
Sep 18 21:52:55.424108 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-image-registry-private-configuration\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.424221 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jhv\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-kube-api-access-j8jhv\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.424221 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-ca-trust-extracted\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.424221 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-installation-pull-secrets\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.424317 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-bound-sa-token\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.424317 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-tls\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.424371 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-trusted-ca\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.424403 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.424387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-certificates\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.463330 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.463311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.463330 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.463320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwtqb" event={"ID":"52d1076e-4c86-496e-a462-154feab9b8b9","Type":"ContainerStarted","Data":"92c8598b9da8e3e4c5d1c5d738d5541a2d2233db66553d47d9df25233f431d20"}
Sep 18 21:52:55.485781 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.485761    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525137 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-trusted-ca\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525230 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-certificates\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525230 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-image-registry-private-configuration\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525230 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525222    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jhv\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-kube-api-access-j8jhv\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525360 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-ca-trust-extracted\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525360 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-installation-pull-secrets\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525360 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-bound-sa-token\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525497 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-tls\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.525653 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-ca-trust-extracted\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.526017 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.525999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-certificates\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.526124 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.526101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-trusted-ca\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.527518 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.527492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-image-registry-private-configuration\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.527606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.527589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-installation-pull-secrets\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.527747 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.527727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-tls\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.534340 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.534312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-bound-sa-token\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.534504 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.534486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jhv\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-kube-api-access-j8jhv\") pod \"image-registry-5655cdf544-z7pmn\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") " pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:55.626282 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-trusted-ca\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.626422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-image-registry-private-configuration\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.626422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-tls\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.626422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626371    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-certificates\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.626422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-ca-trust-extracted\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.626645 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-installation-pull-secrets\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.626707 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:52:55.626841 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626734    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:52:55.626841 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.626821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:52:55.628693 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.628672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:52:55.628836 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.628813    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:52:55.628895 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.628858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:52:55.727577 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8jhv\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-kube-api-access-j8jhv\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.727577 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727530    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-bound-sa-token\") pod \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\" (UID: \"fcfa6ff0-571b-45ad-a5c3-2ef9030efb71\") "
Sep 18 21:52:55.727783 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727734    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-tls\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:55.727783 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727751    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-registry-certificates\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:55.727783 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727766    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-ca-trust-extracted\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:55.727783 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727780    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-installation-pull-secrets\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:55.728016 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727794    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-trusted-ca\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:55.728016 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.727809    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-image-registry-private-configuration\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:55.729613 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.729590    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:52:55.729678 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.729611    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-kube-api-access-j8jhv" (OuterVolumeSpecName: "kube-api-access-j8jhv") pod "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" (UID: "fcfa6ff0-571b-45ad-a5c3-2ef9030efb71"). InnerVolumeSpecName "kube-api-access-j8jhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:52:55.828699 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.828658    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8jhv\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-kube-api-access-j8jhv\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:55.828699 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:55.828696    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71-bound-sa-token\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:52:56.473170 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.473136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwtqb" event={"ID":"52d1076e-4c86-496e-a462-154feab9b8b9","Type":"ContainerStarted","Data":"87d799fc481d85bf56dc691a956be3547ee4faab6603c54e0134d84e35e123a1"}
Sep 18 21:52:56.473568 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.473216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwtqb" event={"ID":"52d1076e-4c86-496e-a462-154feab9b8b9","Type":"ContainerStarted","Data":"067b929bc3d18990e7d89dca0e5114abc201a975b057cecf5118eab24b586523"}
Sep 18 21:52:56.475234 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.475210    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tv5mn" event={"ID":"e3838c43-2bb5-4a08-bde1-9844e7401f3a","Type":"ContainerStarted","Data":"ce0605ef32567742e73c8519ba3ec5843bd82d0f2010b2de94e5481f457a12ab"}
Sep 18 21:52:56.475322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.475244    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5655cdf544-z7pmn"
Sep 18 21:52:56.546289 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.546199    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nwtqb" podStartSLOduration=130.676454325 podStartE2EDuration="2m11.54618037s" totalImagesPullingTime="869.726045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:56.494916856 +0000 UTC m=+133.528095778" watchObservedRunningTime="2026-09-18 21:52:56.54618037 +0000 UTC m=+133.579359263"
Sep 18 21:52:56.566801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.566777    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5655cdf544-z7pmn"]
Sep 18 21:52:56.581500 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.581476    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5655cdf544-z7pmn"]
Sep 18 21:52:56.614714 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.614668    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tv5mn" podStartSLOduration=33.779788407 podStartE2EDuration="35.614649811s" totalImagesPullingTime="1.834861404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 21:52:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:52:56.613907748 +0000 UTC m=+133.647086640" watchObservedRunningTime="2026-09-18 21:52:56.614649811 +0000 UTC m=+133.647828703"
Sep 18 21:52:56.972828 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:56.972792    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcfa6ff0-571b-45ad-a5c3-2ef9030efb71" path="/var/lib/kubelet/pods/fcfa6ff0-571b-45ad-a5c3-2ef9030efb71/volumes"
Sep 18 21:52:58.550095 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:58.550044    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:58.550576 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:58.550121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:52:58.550576 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:52:58.550463    2582 scope.go:117] "RemoveContainer" containerID="0c48c2e8becc3a62ae453647682f760f173d1f4bfd7430bdf097f3644b9a14fb"
Sep 18 21:52:58.550698 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:52:58.550677    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-q4dqp_openshift-console-operator(67298389-9b58-4a22-8f3d-610f24790eb1)\"" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" podUID="67298389-9b58-4a22-8f3d-610f24790eb1"
Sep 18 21:53:00.154657 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:00.154615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:53:00.155397 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:00.155369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3529e86f-3aea-4f0e-9f49-6250abcb042c-service-ca-bundle\") pod \"router-default-657b5f585b-96g68\" (UID: \"3529e86f-3aea-4f0e-9f49-6250abcb042c\") " pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:53:00.440074 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:00.439983    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dhhxf\""
Sep 18 21:53:00.447881 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:00.447861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:53:00.735418 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:00.735326    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-657b5f585b-96g68"]
Sep 18 21:53:00.739107 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:00.739061    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3529e86f_3aea_4f0e_9f49_6250abcb042c.slice/crio-c6e1a151f2c394327689c7c2257bffc697676cbe3b9e2eda6207c64bb5df7d2c WatchSource:0}: Error finding container c6e1a151f2c394327689c7c2257bffc697676cbe3b9e2eda6207c64bb5df7d2c: Status 404 returned error can't find the container with id c6e1a151f2c394327689c7c2257bffc697676cbe3b9e2eda6207c64bb5df7d2c
Sep 18 21:53:01.493762 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:01.493728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-657b5f585b-96g68" event={"ID":"3529e86f-3aea-4f0e-9f49-6250abcb042c","Type":"ContainerStarted","Data":"f24aae00054b001a2c9f040eebbd4296d1b54aa0c2f97949425ec38764db25ee"}
Sep 18 21:53:01.494159 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:01.493776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-657b5f585b-96g68" event={"ID":"3529e86f-3aea-4f0e-9f49-6250abcb042c","Type":"ContainerStarted","Data":"c6e1a151f2c394327689c7c2257bffc697676cbe3b9e2eda6207c64bb5df7d2c"}
Sep 18 21:53:01.518093 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:01.518039    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-657b5f585b-96g68" podStartSLOduration=33.518023337 podStartE2EDuration="33.518023337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:01.51715609 +0000 UTC m=+138.550334984" watchObservedRunningTime="2026-09-18 21:53:01.518023337 +0000 UTC m=+138.551202232"
Sep 18 21:53:02.448284 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:02.448247    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:53:02.450924 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:02.450902    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:53:02.497642 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:02.497615    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:53:02.498706 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:02.498687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-657b5f585b-96g68"
Sep 18 21:53:03.455052 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.455021    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"]
Sep 18 21:53:03.486069 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.486038    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"]
Sep 18 21:53:03.486210 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.486157    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.488315 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.488294    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 21:53:03.489160 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.489142    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h59rg\""
Sep 18 21:53:03.489267 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.489203    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 21:53:03.489267 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.489237    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 21:53:03.489267 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.489242    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 21:53:03.489430 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.489289    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 21:53:03.498458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.498404    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"]
Sep 18 21:53:03.520412 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.520388    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9p8rv"]
Sep 18 21:53:03.520532 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.520500    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.523649 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.523627    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 21:53:03.523878 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.523864    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sczpm\""
Sep 18 21:53:03.524192 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.524171    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 21:53:03.524590 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.524577    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 21:53:03.545821 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.545790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"]
Sep 18 21:53:03.545966 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.545909    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.549583 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.549565    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 21:53:03.549872 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.549845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 21:53:03.549981 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.549897    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 21:53:03.550093 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.550078    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mpctd\""
Sep 18 21:53:03.575362 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.575502 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.575502 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jk8m\" (UniqueName: \"kubernetes.io/projected/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-api-access-7jk8m\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.575502 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575436    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.575502 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-tls\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.575710 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-textfile\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.575710 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.575710 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da788116-d78b-428f-9b38-133b7e5105f5-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.575710 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5j7x\" (UniqueName: \"kubernetes.io/projected/1e49dc96-baa9-43b8-8567-9584ad189e2c-kube-api-access-s5j7x\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.575710 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da788116-d78b-428f-9b38-133b7e5105f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.575980 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-wtmp\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.575980 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da788116-d78b-428f-9b38-133b7e5105f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.575980 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-sys\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.575980 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.575980 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e49dc96-baa9-43b8-8567-9584ad189e2c-metrics-client-ca\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.575980 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.576193 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.575979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7848q\" (UniqueName: \"kubernetes.io/projected/da788116-d78b-428f-9b38-133b7e5105f5-kube-api-access-7848q\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.576193 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.576001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-root\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.576193 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.576018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676387 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.676387 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da788116-d78b-428f-9b38-133b7e5105f5-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.676643 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5j7x\" (UniqueName: \"kubernetes.io/projected/1e49dc96-baa9-43b8-8567-9584ad189e2c-kube-api-access-s5j7x\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676643 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da788116-d78b-428f-9b38-133b7e5105f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.676643 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-wtmp\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676643 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da788116-d78b-428f-9b38-133b7e5105f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.676862 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-sys\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676862 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676672    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-wtmp\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676862 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676862 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e49dc96-baa9-43b8-8567-9584ad189e2c-metrics-client-ca\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676862 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-sys\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.676862 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.677181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7848q\" (UniqueName: \"kubernetes.io/projected/da788116-d78b-428f-9b38-133b7e5105f5-kube-api-access-7848q\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.677181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.676979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-root\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.677181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.677181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.677181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.677181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jk8m\" (UniqueName: \"kubernetes.io/projected/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-api-access-7jk8m\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.677181 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.677513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.677513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-tls\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.677513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da788116-d78b-428f-9b38-133b7e5105f5-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.677513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-textfile\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.677513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.677513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e49dc96-baa9-43b8-8567-9584ad189e2c-metrics-client-ca\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.677513 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e49dc96-baa9-43b8-8567-9584ad189e2c-root\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.677957 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.677957 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.677813    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-textfile\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.678268 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.678240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.679548 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.679523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.679692 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.679661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da788116-d78b-428f-9b38-133b7e5105f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.679847 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.679822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e49dc96-baa9-43b8-8567-9584ad189e2c-node-exporter-tls\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.679922 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.679870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.680174 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.680157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da788116-d78b-428f-9b38-133b7e5105f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.680304 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.680288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.684904 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.684887    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5j7x\" (UniqueName: \"kubernetes.io/projected/1e49dc96-baa9-43b8-8567-9584ad189e2c-kube-api-access-s5j7x\") pod \"node-exporter-9p8rv\" (UID: \"1e49dc96-baa9-43b8-8567-9584ad189e2c\") " pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.688994 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.688975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jk8m\" (UniqueName: \"kubernetes.io/projected/15aebc2b-a06e-4c9f-ac52-24c8158b2f62-kube-api-access-7jk8m\") pod \"kube-state-metrics-84c885658c-cb4bk\" (UID: \"15aebc2b-a06e-4c9f-ac52-24c8158b2f62\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.689380 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.689363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7848q\" (UniqueName: \"kubernetes.io/projected/da788116-d78b-428f-9b38-133b7e5105f5-kube-api-access-7848q\") pod \"openshift-state-metrics-86c5bc79d6-qhv2n\" (UID: \"da788116-d78b-428f-9b38-133b7e5105f5\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.794972 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.794878    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"
Sep 18 21:53:03.829960 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.829908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"
Sep 18 21:53:03.854356 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:03.854245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9p8rv"
Sep 18 21:53:03.865736 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:03.865708    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e49dc96_baa9_43b8_8567_9584ad189e2c.slice/crio-fadd565c71b6764086dd8676745776cef0c310ade035cb9ecccd4ad3f8edf89b WatchSource:0}: Error finding container fadd565c71b6764086dd8676745776cef0c310ade035cb9ecccd4ad3f8edf89b: Status 404 returned error can't find the container with id fadd565c71b6764086dd8676745776cef0c310ade035cb9ecccd4ad3f8edf89b
Sep 18 21:53:04.076094 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:04.072423    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n"]
Sep 18 21:53:04.122643 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:04.122530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-cb4bk"]
Sep 18 21:53:04.124429 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:04.124404    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15aebc2b_a06e_4c9f_ac52_24c8158b2f62.slice/crio-2b9de6fe5b361a8941766764c7ee51e0f0693531ab34846bf985a727c8f91a22 WatchSource:0}: Error finding container 2b9de6fe5b361a8941766764c7ee51e0f0693531ab34846bf985a727c8f91a22: Status 404 returned error can't find the container with id 2b9de6fe5b361a8941766764c7ee51e0f0693531ab34846bf985a727c8f91a22
Sep 18 21:53:04.504872 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:04.504839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9p8rv" event={"ID":"1e49dc96-baa9-43b8-8567-9584ad189e2c","Type":"ContainerStarted","Data":"fadd565c71b6764086dd8676745776cef0c310ade035cb9ecccd4ad3f8edf89b"}
Sep 18 21:53:04.505972 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:04.505952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk" event={"ID":"15aebc2b-a06e-4c9f-ac52-24c8158b2f62","Type":"ContainerStarted","Data":"2b9de6fe5b361a8941766764c7ee51e0f0693531ab34846bf985a727c8f91a22"}
Sep 18 21:53:04.507600 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:04.507578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n" event={"ID":"da788116-d78b-428f-9b38-133b7e5105f5","Type":"ContainerStarted","Data":"c3ecb74ab4d5109ebb08a7006b586f64498e1ebc70aaebcd500715928cabcedc"}
Sep 18 21:53:04.507677 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:04.507616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n" event={"ID":"da788116-d78b-428f-9b38-133b7e5105f5","Type":"ContainerStarted","Data":"c40bc2005a57d98d2050ec59aa2899db70b18c17cbf2e7ae0a383f4550de3e82"}
Sep 18 21:53:04.507677 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:04.507634    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n" event={"ID":"da788116-d78b-428f-9b38-133b7e5105f5","Type":"ContainerStarted","Data":"6cf51f6b636afc1698b1d09b45de0c84261e2e3dedd074a3c4108400e9da13d5"}
Sep 18 21:53:05.513678 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.513642    2582 generic.go:356] "Generic (PLEG): container finished" podID="1e49dc96-baa9-43b8-8567-9584ad189e2c" containerID="ada62e377367cbd0b1fde7d78df3da500c9f9aa14b303ae4bed61a4ac7e5ada5" exitCode=0
Sep 18 21:53:05.514179 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.513705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9p8rv" event={"ID":"1e49dc96-baa9-43b8-8567-9584ad189e2c","Type":"ContainerDied","Data":"ada62e377367cbd0b1fde7d78df3da500c9f9aa14b303ae4bed61a4ac7e5ada5"}
Sep 18 21:53:05.595370 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.595340    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"]
Sep 18 21:53:05.599340 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.599317    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.601926 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.601898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 21:53:05.602051 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.602009    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 21:53:05.602122 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.602076    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h5pgq\""
Sep 18 21:53:05.602606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.602584    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 21:53:05.602733 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.602604    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 21:53:05.603080 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.603061    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 21:53:05.603166 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.603103    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5dle1q70h381g\""
Sep 18 21:53:05.613940 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.613915    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"]
Sep 18 21:53:05.691847 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.691820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.691937 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.691875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56c5\" (UniqueName: \"kubernetes.io/projected/6d47c0e8-a685-4bc9-a1a1-68e770448194-kube-api-access-h56c5\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.691937 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.691919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-grpc-tls\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.692046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.691971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-tls\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.692046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.691989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d47c0e8-a685-4bc9-a1a1-68e770448194-metrics-client-ca\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.692046 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.692007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.692263 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.692043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.692263 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.692069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.792829 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.792804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.792924 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.792842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.792924 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.792874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.792924 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.792911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56c5\" (UniqueName: \"kubernetes.io/projected/6d47c0e8-a685-4bc9-a1a1-68e770448194-kube-api-access-h56c5\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.793550 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.793154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-grpc-tls\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.793550 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.793198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-tls\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.793550 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.793215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d47c0e8-a685-4bc9-a1a1-68e770448194-metrics-client-ca\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.793550 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.793238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.794063 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.793988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d47c0e8-a685-4bc9-a1a1-68e770448194-metrics-client-ca\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.795841 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.795815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.795962 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.795859    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-tls\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.796320 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.796300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.796388 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.796329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.796388 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.796371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-grpc-tls\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.796812 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.796478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6d47c0e8-a685-4bc9-a1a1-68e770448194-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.804676 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.804030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56c5\" (UniqueName: \"kubernetes.io/projected/6d47c0e8-a685-4bc9-a1a1-68e770448194-kube-api-access-h56c5\") pod \"thanos-querier-5c9d7f8c77-5p257\" (UID: \"6d47c0e8-a685-4bc9-a1a1-68e770448194\") " pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:05.910826 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:05.910801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:06.205365 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.205333    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"]
Sep 18 21:53:06.208747 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:06.208721    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d47c0e8_a685_4bc9_a1a1_68e770448194.slice/crio-d0b6e2a2423234b37ed5b75a5bfef4c9e66fb94683732e56642b31bc4bc97013 WatchSource:0}: Error finding container d0b6e2a2423234b37ed5b75a5bfef4c9e66fb94683732e56642b31bc4bc97013: Status 404 returned error can't find the container with id d0b6e2a2423234b37ed5b75a5bfef4c9e66fb94683732e56642b31bc4bc97013
Sep 18 21:53:06.520572 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.520479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9p8rv" event={"ID":"1e49dc96-baa9-43b8-8567-9584ad189e2c","Type":"ContainerStarted","Data":"1251ef1c76dfcf68b1f1c89373fa0b7c9b756bfdad333ac5a3a024a90a374bcc"}
Sep 18 21:53:06.520572 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.520537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9p8rv" event={"ID":"1e49dc96-baa9-43b8-8567-9584ad189e2c","Type":"ContainerStarted","Data":"09e73513860d2d7c7f341d89358534b92b6c0984b6b0a09b9ebdabff995ba942"}
Sep 18 21:53:06.522624 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.522599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk" event={"ID":"15aebc2b-a06e-4c9f-ac52-24c8158b2f62","Type":"ContainerStarted","Data":"e7d7d1da00f8ae12d0e2a5b9897e0729a948b4b56796c5025396c21c7e16a1c9"}
Sep 18 21:53:06.522748 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.522655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk" event={"ID":"15aebc2b-a06e-4c9f-ac52-24c8158b2f62","Type":"ContainerStarted","Data":"381fe0c9270f0deb121090bce9f048b0a2d7e3374ec45fdc5940008b1c656785"}
Sep 18 21:53:06.522748 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.522684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk" event={"ID":"15aebc2b-a06e-4c9f-ac52-24c8158b2f62","Type":"ContainerStarted","Data":"b0d8031cc78d42fbb5c7d90c0fde3e38ce011bcb1c079355208e9a9b62c9df6d"}
Sep 18 21:53:06.524617 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.524595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n" event={"ID":"da788116-d78b-428f-9b38-133b7e5105f5","Type":"ContainerStarted","Data":"44477dc80cdb02d7dd1b38c86d2a74afe111af01e59eed1a9c2c6f47f31bc088"}
Sep 18 21:53:06.525738 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.525713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" event={"ID":"6d47c0e8-a685-4bc9-a1a1-68e770448194","Type":"ContainerStarted","Data":"d0b6e2a2423234b37ed5b75a5bfef4c9e66fb94683732e56642b31bc4bc97013"}
Sep 18 21:53:06.545588 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.545536    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9p8rv" podStartSLOduration=2.5935313129999997 podStartE2EDuration="3.545518442s" totalImagesPullingTime="884.966609ms" totalInitContainerRuntime="67.02052ms" isStatefulPod=true podCreationTimestamp="2026-09-18 21:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:06.544013846 +0000 UTC m=+143.577192740" watchObservedRunningTime="2026-09-18 21:53:06.545518442 +0000 UTC m=+143.578697360"
Sep 18 21:53:06.563488 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.563437    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qhv2n" podStartSLOduration=2.078790669 podStartE2EDuration="3.56342151s" totalImagesPullingTime="1.484630841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:06.562110118 +0000 UTC m=+143.595289012" watchObservedRunningTime="2026-09-18 21:53:06.56342151 +0000 UTC m=+143.596600404"
Sep 18 21:53:06.583052 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:06.582999    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-cb4bk" podStartSLOduration=2.025639879 podStartE2EDuration="3.582981033s" totalImagesPullingTime="1.557341154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:06.582164551 +0000 UTC m=+143.615343450" watchObservedRunningTime="2026-09-18 21:53:06.582981033 +0000 UTC m=+143.616159924"
Sep 18 21:53:08.221989 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.221939    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-492lt"]
Sep 18 21:53:08.225475 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.225449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:08.229427 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.229400    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r9qn6\""
Sep 18 21:53:08.229637 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.229621    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 21:53:08.237064 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.237041    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-492lt"]
Sep 18 21:53:08.316793 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.316706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:08.316793 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.316743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lln\" (UniqueName: \"kubernetes.io/projected/702652c4-e8c4-4692-bcbc-14728c18b1e4-kube-api-access-98lln\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:08.417872 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.417834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:08.418069 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.417881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lln\" (UniqueName: \"kubernetes.io/projected/702652c4-e8c4-4692-bcbc-14728c18b1e4-kube-api-access-98lln\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:08.418069 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:53:08.417977    2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 18 21:53:08.418069 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:53:08.418046    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert podName:702652c4-e8c4-4692-bcbc-14728c18b1e4 nodeName:}" failed. No retries permitted until 2026-09-18 21:53:08.918026708 +0000 UTC m=+145.951205583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-492lt" (UID: "702652c4-e8c4-4692-bcbc-14728c18b1e4") : secret "monitoring-plugin-cert" not found
Sep 18 21:53:08.429466 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.429436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lln\" (UniqueName: \"kubernetes.io/projected/702652c4-e8c4-4692-bcbc-14728c18b1e4-kube-api-access-98lln\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:08.535081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.535044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" event={"ID":"6d47c0e8-a685-4bc9-a1a1-68e770448194","Type":"ContainerStarted","Data":"56a77275801e03966194ac733b123f9c6b6187c0d8a59e1e9870ff0487a8c8e8"}
Sep 18 21:53:08.535231 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.535113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" event={"ID":"6d47c0e8-a685-4bc9-a1a1-68e770448194","Type":"ContainerStarted","Data":"8be6d0a25407d98a8135a5b5b059a93787cd0a58412678bad8e0cb1ee105d6b1"}
Sep 18 21:53:08.535231 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.535139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" event={"ID":"6d47c0e8-a685-4bc9-a1a1-68e770448194","Type":"ContainerStarted","Data":"6d073faac3faf9c3066aa97bba38a39469093f295ac73e9e4ddeb8afbbdfb01c"}
Sep 18 21:53:08.921336 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:08.921295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:08.921516 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:53:08.921486    2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 18 21:53:08.921586 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:53:08.921574    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert podName:702652c4-e8c4-4692-bcbc-14728c18b1e4 nodeName:}" failed. No retries permitted until 2026-09-18 21:53:09.921551811 +0000 UTC m=+146.954730698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-492lt" (UID: "702652c4-e8c4-4692-bcbc-14728c18b1e4") : secret "monitoring-plugin-cert" not found
Sep 18 21:53:09.541489 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:09.541393    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" event={"ID":"6d47c0e8-a685-4bc9-a1a1-68e770448194","Type":"ContainerStarted","Data":"75902ec5ceb75209318e45df8b00ec301d89169f68b7eb2923e1f431a8ba2d86"}
Sep 18 21:53:09.541489 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:09.541462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" event={"ID":"6d47c0e8-a685-4bc9-a1a1-68e770448194","Type":"ContainerStarted","Data":"810ca596be0a39c2a9d35d63c39198b8617c3284e0dd1f558d1931d8744cbf65"}
Sep 18 21:53:09.541489 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:09.541490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" event={"ID":"6d47c0e8-a685-4bc9-a1a1-68e770448194","Type":"ContainerStarted","Data":"0db605f322fc5b491c9dbb1ba5cf73f7d6b9fd1c53ec4179b4f8f2a28d5f1a5d"}
Sep 18 21:53:09.541920 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:09.541533    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:09.567809 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:09.567761    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257" podStartSLOduration=1.956914141 podStartE2EDuration="4.567744489s" totalImagesPullingTime="2.610830348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:09.565453425 +0000 UTC m=+146.598632316" watchObservedRunningTime="2026-09-18 21:53:09.567744489 +0000 UTC m=+146.600923382"
Sep 18 21:53:09.929075 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:09.929033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:09.931422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:09.931399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/702652c4-e8c4-4692-bcbc-14728c18b1e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-492lt\" (UID: \"702652c4-e8c4-4692-bcbc-14728c18b1e4\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:10.035330 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:10.035290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:10.308251 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:10.308175    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-492lt"]
Sep 18 21:53:10.311588 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:10.311559    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702652c4_e8c4_4692_bcbc_14728c18b1e4.slice/crio-423638c34048e702d486f37980632093fa3de63c33d08111eb7c326c6af5be1d WatchSource:0}: Error finding container 423638c34048e702d486f37980632093fa3de63c33d08111eb7c326c6af5be1d: Status 404 returned error can't find the container with id 423638c34048e702d486f37980632093fa3de63c33d08111eb7c326c6af5be1d
Sep 18 21:53:10.546357 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:10.546319    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt" event={"ID":"702652c4-e8c4-4692-bcbc-14728c18b1e4","Type":"ContainerStarted","Data":"423638c34048e702d486f37980632093fa3de63c33d08111eb7c326c6af5be1d"}
Sep 18 21:53:10.969253 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:10.969222    2582 scope.go:117] "RemoveContainer" containerID="0c48c2e8becc3a62ae453647682f760f173d1f4bfd7430bdf097f3644b9a14fb"
Sep 18 21:53:11.552117 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.552087    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 21:53:11.552497 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.552139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" event={"ID":"67298389-9b58-4a22-8f3d-610f24790eb1","Type":"ContainerStarted","Data":"2317bad966638a2e12f680fb9803d46686f697edb6ecd0a42a5b7c8bdefb7322"}
Sep 18 21:53:11.552497 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.552424    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:53:11.588840 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.588788    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp" podStartSLOduration=40.045822838 podStartE2EDuration="43.588768804s" totalImagesPullingTime="3.542945966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:11.588543307 +0000 UTC m=+148.621722202" watchObservedRunningTime="2026-09-18 21:53:11.588768804 +0000 UTC m=+148.621947700"
Sep 18 21:53:11.649022 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.649000    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-q4dqp"
Sep 18 21:53:11.937798 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.937760    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-bgdjw"]
Sep 18 21:53:11.941061 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.941037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-bgdjw"
Sep 18 21:53:11.943590 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.943566    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 21:53:11.944065 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.944050    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 21:53:11.944148 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.944068    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hfz2w\""
Sep 18 21:53:11.960682 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:11.960653    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-bgdjw"]
Sep 18 21:53:12.050342 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.050313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/99f0f008-3991-4448-9772-bf0266a9919e-kube-api-access-bsnw6\") pod \"downloads-6594f4c75-bgdjw\" (UID: \"99f0f008-3991-4448-9772-bf0266a9919e\") " pod="openshift-console/downloads-6594f4c75-bgdjw"
Sep 18 21:53:12.151548 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.151520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/99f0f008-3991-4448-9772-bf0266a9919e-kube-api-access-bsnw6\") pod \"downloads-6594f4c75-bgdjw\" (UID: \"99f0f008-3991-4448-9772-bf0266a9919e\") " pod="openshift-console/downloads-6594f4c75-bgdjw"
Sep 18 21:53:12.165915 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.165888    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnw6\" (UniqueName: \"kubernetes.io/projected/99f0f008-3991-4448-9772-bf0266a9919e-kube-api-access-bsnw6\") pod \"downloads-6594f4c75-bgdjw\" (UID: \"99f0f008-3991-4448-9772-bf0266a9919e\") " pod="openshift-console/downloads-6594f4c75-bgdjw"
Sep 18 21:53:12.249708 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.249631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-bgdjw"
Sep 18 21:53:12.523598 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.523516    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-bgdjw"]
Sep 18 21:53:12.526373 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:12.526343    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f0f008_3991_4448_9772_bf0266a9919e.slice/crio-88e2ba6c760e6cd1087c3b633ee5cc2571f0dd736d5500265ea206d75ec4693c WatchSource:0}: Error finding container 88e2ba6c760e6cd1087c3b633ee5cc2571f0dd736d5500265ea206d75ec4693c: Status 404 returned error can't find the container with id 88e2ba6c760e6cd1087c3b633ee5cc2571f0dd736d5500265ea206d75ec4693c
Sep 18 21:53:12.556825 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.556793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-bgdjw" event={"ID":"99f0f008-3991-4448-9772-bf0266a9919e","Type":"ContainerStarted","Data":"88e2ba6c760e6cd1087c3b633ee5cc2571f0dd736d5500265ea206d75ec4693c"}
Sep 18 21:53:12.558264 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.558238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt" event={"ID":"702652c4-e8c4-4692-bcbc-14728c18b1e4","Type":"ContainerStarted","Data":"7c728a98a6f363e0011321811b36380d233b3ffd72dded40197f933a0541841b"}
Sep 18 21:53:12.558489 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.558465    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:12.562908 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.562888    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt"
Sep 18 21:53:12.582715 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:12.582670    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-492lt" podStartSLOduration=3.308129209 podStartE2EDuration="4.582654783s" totalImagesPullingTime="1.274525574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:12.580472596 +0000 UTC m=+149.613651501" watchObservedRunningTime="2026-09-18 21:53:12.582654783 +0000 UTC m=+149.615833678"
Sep 18 21:53:15.552643 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:15.552612    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c9d7f8c77-5p257"
Sep 18 21:53:18.258680 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.258641    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bd5655748-mtdd5"]
Sep 18 21:53:18.262624 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.262600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.266561 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.266465    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 21:53:18.266561 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.266482    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 21:53:18.266894 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.266872    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 21:53:18.266992 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.266901    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9nz2c\""
Sep 18 21:53:18.266992 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.266919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 21:53:18.267143 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.267050    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 21:53:18.276172 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.276128    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd5655748-mtdd5"]
Sep 18 21:53:18.304413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.304380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-serving-cert\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.304413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.304417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-service-ca\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.304611 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.304437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-oauth-config\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.304611 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.304547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7p66\" (UniqueName: \"kubernetes.io/projected/ee1ba6b0-5136-48ff-bf39-27220cf86acf-kube-api-access-s7p66\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.304694 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.304631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-oauth-serving-cert\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.304694 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.304666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-config\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.406002 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.405961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-serving-cert\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.406002 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.406005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-service-ca\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.406215 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.406037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-oauth-config\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.406215 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.406076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7p66\" (UniqueName: \"kubernetes.io/projected/ee1ba6b0-5136-48ff-bf39-27220cf86acf-kube-api-access-s7p66\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.406215 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.406148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-oauth-serving-cert\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.406215 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.406177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-config\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.406894 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.406864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-service-ca\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.407051 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.406900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-config\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.407051 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.407025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-oauth-serving-cert\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.408729 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.408704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-oauth-config\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.408838 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.408778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-serving-cert\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.414613 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.414581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7p66\" (UniqueName: \"kubernetes.io/projected/ee1ba6b0-5136-48ff-bf39-27220cf86acf-kube-api-access-s7p66\") pod \"console-7bd5655748-mtdd5\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") " pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.575797 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.575703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:18.883675 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:18.883641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd5655748-mtdd5"]
Sep 18 21:53:18.887603 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:18.887570    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1ba6b0_5136_48ff_bf39_27220cf86acf.slice/crio-b3aeb8f6f34f6a07babe9c7862bf7b6291c9fbacfdf707e5213f4044b100b2ed WatchSource:0}: Error finding container b3aeb8f6f34f6a07babe9c7862bf7b6291c9fbacfdf707e5213f4044b100b2ed: Status 404 returned error can't find the container with id b3aeb8f6f34f6a07babe9c7862bf7b6291c9fbacfdf707e5213f4044b100b2ed
Sep 18 21:53:19.595183 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:19.595141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5655748-mtdd5" event={"ID":"ee1ba6b0-5136-48ff-bf39-27220cf86acf","Type":"ContainerStarted","Data":"b3aeb8f6f34f6a07babe9c7862bf7b6291c9fbacfdf707e5213f4044b100b2ed"}
Sep 18 21:53:21.119207 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:53:21.119159    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-c4tmv" podUID="9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c"
Sep 18 21:53:21.606741 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:21.606664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:53:25.982211 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:25.982172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:53:25.984687 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:25.984660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c-metrics-tls\") pod \"dns-default-c4tmv\" (UID: \"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c\") " pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:53:26.112102 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:26.112067    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gb4w8\""
Sep 18 21:53:26.118191 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:26.118162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:53:30.651533 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:30.651503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5655748-mtdd5" event={"ID":"ee1ba6b0-5136-48ff-bf39-27220cf86acf","Type":"ContainerStarted","Data":"731ca5262744871a21016874bf524f7a60497a09bb26c32359543460ae821935"}
Sep 18 21:53:30.789598 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:30.789567    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4tmv"]
Sep 18 21:53:30.791865 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:53:30.791837    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebd9ea5_1ea3_46c5_b03b_4c0224a1b68c.slice/crio-fc4875f06149f4d214f686768c0fa5417317a62258a912b6e87049a58f27c9b6 WatchSource:0}: Error finding container fc4875f06149f4d214f686768c0fa5417317a62258a912b6e87049a58f27c9b6: Status 404 returned error can't find the container with id fc4875f06149f4d214f686768c0fa5417317a62258a912b6e87049a58f27c9b6
Sep 18 21:53:31.659111 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:31.659071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-bgdjw" event={"ID":"99f0f008-3991-4448-9772-bf0266a9919e","Type":"ContainerStarted","Data":"b73b0052de2b52b8d02a2a8b2f72f653387692f23e6db20067e6afd3a6f3c730"}
Sep 18 21:53:31.659659 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:31.659590    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-bgdjw"
Sep 18 21:53:31.661734 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:31.661470    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4tmv" event={"ID":"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c","Type":"ContainerStarted","Data":"fc4875f06149f4d214f686768c0fa5417317a62258a912b6e87049a58f27c9b6"}
Sep 18 21:53:31.683030 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:31.682919    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-bgdjw" podStartSLOduration=2.635306743 podStartE2EDuration="20.682898017s" totalImagesPullingTime="18.047591274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:31.682069432 +0000 UTC m=+168.715248328" watchObservedRunningTime="2026-09-18 21:53:31.682898017 +0000 UTC m=+168.716076912"
Sep 18 21:53:31.683906 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:31.683866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-bgdjw"
Sep 18 21:53:31.706507 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:31.706446    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bd5655748-mtdd5" podStartSLOduration=2.080037299 podStartE2EDuration="13.706424784s" totalImagesPullingTime="11.626387485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:31.70417647 +0000 UTC m=+168.737355365" watchObservedRunningTime="2026-09-18 21:53:31.706424784 +0000 UTC m=+168.739603680"
Sep 18 21:53:33.672796 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:33.672730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4tmv" event={"ID":"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c","Type":"ContainerStarted","Data":"0acfc0c895c738c90e01cfcfe78976b7798c7f768fbb01b1089548ce4d94fc36"}
Sep 18 21:53:33.673260 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:33.672829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4tmv" event={"ID":"9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c","Type":"ContainerStarted","Data":"0258d9616575a79322cb49ad22ebd8de9a96337bec35951662b97e3afe319baa"}
Sep 18 21:53:33.698027 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:33.697905    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c4tmv" podStartSLOduration=133.304413012 podStartE2EDuration="2m15.697881418s" totalImagesPullingTime="2.393468406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:53:33.696496338 +0000 UTC m=+170.729675235" watchObservedRunningTime="2026-09-18 21:53:33.697881418 +0000 UTC m=+170.731060315"
Sep 18 21:53:34.677252 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:34.677206    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:53:35.520580 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:35.520541    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd5655748-mtdd5"]
Sep 18 21:53:38.576502 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:38.576468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:53:44.684485 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:53:44.684455    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c4tmv"
Sep 18 21:54:00.543012 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.542966    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bd5655748-mtdd5" podUID="ee1ba6b0-5136-48ff-bf39-27220cf86acf" containerName="console" containerID="cri-o://731ca5262744871a21016874bf524f7a60497a09bb26c32359543460ae821935" gracePeriod=15
Sep 18 21:54:00.784514 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.784482    2582 generic.go:356] "Generic (PLEG): container finished" podID="ee1ba6b0-5136-48ff-bf39-27220cf86acf" containerID="731ca5262744871a21016874bf524f7a60497a09bb26c32359543460ae821935" exitCode=0
Sep 18 21:54:00.784628 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.784545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5655748-mtdd5" event={"ID":"ee1ba6b0-5136-48ff-bf39-27220cf86acf","Type":"ContainerDied","Data":"731ca5262744871a21016874bf524f7a60497a09bb26c32359543460ae821935"}
Sep 18 21:54:00.822759 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.822689    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:54:00.873635 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.873611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-oauth-serving-cert\") pod \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") "
Sep 18 21:54:00.873801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.873654    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-serving-cert\") pod \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") "
Sep 18 21:54:00.873801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.873684    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-service-ca\") pod \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") "
Sep 18 21:54:00.873801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.873703    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-oauth-config\") pod \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") "
Sep 18 21:54:00.873801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.873724    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7p66\" (UniqueName: \"kubernetes.io/projected/ee1ba6b0-5136-48ff-bf39-27220cf86acf-kube-api-access-s7p66\") pod \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") "
Sep 18 21:54:00.873801 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.873752    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-config\") pod \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\" (UID: \"ee1ba6b0-5136-48ff-bf39-27220cf86acf\") "
Sep 18 21:54:00.874078 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.873975    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee1ba6b0-5136-48ff-bf39-27220cf86acf" (UID: "ee1ba6b0-5136-48ff-bf39-27220cf86acf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:00.874223 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.874192    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee1ba6b0-5136-48ff-bf39-27220cf86acf" (UID: "ee1ba6b0-5136-48ff-bf39-27220cf86acf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:00.874324 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.874239    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-config" (OuterVolumeSpecName: "console-config") pod "ee1ba6b0-5136-48ff-bf39-27220cf86acf" (UID: "ee1ba6b0-5136-48ff-bf39-27220cf86acf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 21:54:00.875932 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.875908    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee1ba6b0-5136-48ff-bf39-27220cf86acf" (UID: "ee1ba6b0-5136-48ff-bf39-27220cf86acf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:00.876061 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.876037    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1ba6b0-5136-48ff-bf39-27220cf86acf-kube-api-access-s7p66" (OuterVolumeSpecName: "kube-api-access-s7p66") pod "ee1ba6b0-5136-48ff-bf39-27220cf86acf" (UID: "ee1ba6b0-5136-48ff-bf39-27220cf86acf"). InnerVolumeSpecName "kube-api-access-s7p66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:54:00.876110 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.876094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee1ba6b0-5136-48ff-bf39-27220cf86acf" (UID: "ee1ba6b0-5136-48ff-bf39-27220cf86acf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 21:54:00.974751 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.974722    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7p66\" (UniqueName: \"kubernetes.io/projected/ee1ba6b0-5136-48ff-bf39-27220cf86acf-kube-api-access-s7p66\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:54:00.974751 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.974748    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-config\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:54:00.974922 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.974757    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-oauth-serving-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:54:00.974922 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.974766    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-serving-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:54:00.974922 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.974775    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1ba6b0-5136-48ff-bf39-27220cf86acf-service-ca\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:54:00.974922 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:00.974788    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee1ba6b0-5136-48ff-bf39-27220cf86acf-console-oauth-config\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:54:01.789573 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:01.789543    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5655748-mtdd5"
Sep 18 21:54:01.790006 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:01.789536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5655748-mtdd5" event={"ID":"ee1ba6b0-5136-48ff-bf39-27220cf86acf","Type":"ContainerDied","Data":"b3aeb8f6f34f6a07babe9c7862bf7b6291c9fbacfdf707e5213f4044b100b2ed"}
Sep 18 21:54:01.790006 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:01.789673    2582 scope.go:117] "RemoveContainer" containerID="731ca5262744871a21016874bf524f7a60497a09bb26c32359543460ae821935"
Sep 18 21:54:01.810535 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:01.810512    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd5655748-mtdd5"]
Sep 18 21:54:01.817163 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:01.817140    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bd5655748-mtdd5"]
Sep 18 21:54:02.795186 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:02.795108    2582 generic.go:356] "Generic (PLEG): container finished" podID="3e494d22-0a87-4169-ac16-40037cdeaee6" containerID="11d34dfcdfe4666fbd601889e10d8326535cf307fa8efeecd3531e9b85dc4267" exitCode=0
Sep 18 21:54:02.795564 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:02.795187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p" event={"ID":"3e494d22-0a87-4169-ac16-40037cdeaee6","Type":"ContainerDied","Data":"11d34dfcdfe4666fbd601889e10d8326535cf307fa8efeecd3531e9b85dc4267"}
Sep 18 21:54:02.795564 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:02.795541    2582 scope.go:117] "RemoveContainer" containerID="11d34dfcdfe4666fbd601889e10d8326535cf307fa8efeecd3531e9b85dc4267"
Sep 18 21:54:02.974345 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:02.974298    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1ba6b0-5136-48ff-bf39-27220cf86acf" path="/var/lib/kubelet/pods/ee1ba6b0-5136-48ff-bf39-27220cf86acf/volumes"
Sep 18 21:54:03.802287 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:03.802253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vl67p" event={"ID":"3e494d22-0a87-4169-ac16-40037cdeaee6","Type":"ContainerStarted","Data":"f233a3de129980d1915d68e1180d3b998a534096c136bf0df2e626d730140077"}
Sep 18 21:54:07.819659 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:07.819621    2582 generic.go:356] "Generic (PLEG): container finished" podID="6f64f8e8-176b-493f-8839-e1d04f98be13" containerID="4eb7cf157947c24f71671bb9e797ad7e9e87bcaf3fd59b9a63640de3c9f67646" exitCode=0
Sep 18 21:54:07.820104 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:07.819686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl" event={"ID":"6f64f8e8-176b-493f-8839-e1d04f98be13","Type":"ContainerDied","Data":"4eb7cf157947c24f71671bb9e797ad7e9e87bcaf3fd59b9a63640de3c9f67646"}
Sep 18 21:54:07.820104 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:07.820042    2582 scope.go:117] "RemoveContainer" containerID="4eb7cf157947c24f71671bb9e797ad7e9e87bcaf3fd59b9a63640de3c9f67646"
Sep 18 21:54:08.826710 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:54:08.826677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-46bsl" event={"ID":"6f64f8e8-176b-493f-8839-e1d04f98be13","Type":"ContainerStarted","Data":"c8706f5642cf7b1fd613ac848ae529b0d4324733bd94c995b01d71247113bf06"}
Sep 18 21:55:44.850914 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:55:44.850885    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 21:55:44.852557 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:55:44.852537    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 21:55:44.857117 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:55:44.857095    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 21:55:44.858620 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:55:44.858600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 21:55:44.862931 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:55:44.862917    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 21:56:51.507572 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.507537    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"]
Sep 18 21:56:51.508024 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.507846    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1ba6b0-5136-48ff-bf39-27220cf86acf" containerName="console"
Sep 18 21:56:51.508024 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.507858    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1ba6b0-5136-48ff-bf39-27220cf86acf" containerName="console"
Sep 18 21:56:51.508024 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.507908    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1ba6b0-5136-48ff-bf39-27220cf86acf" containerName="console"
Sep 18 21:56:51.511032 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.511011    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.514677 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.514651    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 21:56:51.514784 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.514651    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ggxp8\""
Sep 18 21:56:51.515178 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.515157    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 21:56:51.527138 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.527112    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"]
Sep 18 21:56:51.559610 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.559584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.559714 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.559623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.559714 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.559693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7265z\" (UniqueName: \"kubernetes.io/projected/bee8c9e7-c721-40d5-951c-81973bd7b143-kube-api-access-7265z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.660381 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.660350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7265z\" (UniqueName: \"kubernetes.io/projected/bee8c9e7-c721-40d5-951c-81973bd7b143-kube-api-access-7265z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.660500 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.660397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.660500 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.660425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.660796 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.660778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.660870 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.660808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.672678 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.672650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7265z\" (UniqueName: \"kubernetes.io/projected/bee8c9e7-c721-40d5-951c-81973bd7b143-kube-api-access-7265z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:51.819976 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:51.819854    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:56:52.092722 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:52.092500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"]
Sep 18 21:56:52.095245 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:56:52.095216    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee8c9e7_c721_40d5_951c_81973bd7b143.slice/crio-13745ccde3aeeb71f076cd8ce50a739b05d7d84c1b11f5f5cc4fb72eeb70b7e1 WatchSource:0}: Error finding container 13745ccde3aeeb71f076cd8ce50a739b05d7d84c1b11f5f5cc4fb72eeb70b7e1: Status 404 returned error can't find the container with id 13745ccde3aeeb71f076cd8ce50a739b05d7d84c1b11f5f5cc4fb72eeb70b7e1
Sep 18 21:56:52.097337 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:52.097320    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 21:56:52.464865 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:52.464829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq" event={"ID":"bee8c9e7-c721-40d5-951c-81973bd7b143","Type":"ContainerStarted","Data":"13745ccde3aeeb71f076cd8ce50a739b05d7d84c1b11f5f5cc4fb72eeb70b7e1"}
Sep 18 21:56:58.492585 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:58.492549    2582 generic.go:356] "Generic (PLEG): container finished" podID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerID="03a3aaf7986862c1db54ec370d871f6d5bfaa6ac56caecc241aa19614bd7ecc2" exitCode=0
Sep 18 21:56:58.492999 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:56:58.492628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq" event={"ID":"bee8c9e7-c721-40d5-951c-81973bd7b143","Type":"ContainerDied","Data":"03a3aaf7986862c1db54ec370d871f6d5bfaa6ac56caecc241aa19614bd7ecc2"}
Sep 18 21:57:01.508004 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:01.507920    2582 generic.go:356] "Generic (PLEG): container finished" podID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerID="77799db75cb52f58444744be702ceb42ce608aaa13066eac4f9e6b6d7b464e47" exitCode=0
Sep 18 21:57:01.508378 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:01.508017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq" event={"ID":"bee8c9e7-c721-40d5-951c-81973bd7b143","Type":"ContainerDied","Data":"77799db75cb52f58444744be702ceb42ce608aaa13066eac4f9e6b6d7b464e47"}
Sep 18 21:57:09.546431 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:09.546393    2582 generic.go:356] "Generic (PLEG): container finished" podID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerID="bf578ac4b82e019de459b328ae07ca4f834bc5cf99856f09ddac403291c26cd1" exitCode=0
Sep 18 21:57:09.546800 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:09.546458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq" event={"ID":"bee8c9e7-c721-40d5-951c-81973bd7b143","Type":"ContainerDied","Data":"bf578ac4b82e019de459b328ae07ca4f834bc5cf99856f09ddac403291c26cd1"}
Sep 18 21:57:10.668809 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.668786    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:57:10.824153 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.824069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7265z\" (UniqueName: \"kubernetes.io/projected/bee8c9e7-c721-40d5-951c-81973bd7b143-kube-api-access-7265z\") pod \"bee8c9e7-c721-40d5-951c-81973bd7b143\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") "
Sep 18 21:57:10.824153 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.824132    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-util\") pod \"bee8c9e7-c721-40d5-951c-81973bd7b143\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") "
Sep 18 21:57:10.824367 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.824191    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-bundle\") pod \"bee8c9e7-c721-40d5-951c-81973bd7b143\" (UID: \"bee8c9e7-c721-40d5-951c-81973bd7b143\") "
Sep 18 21:57:10.824995 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.824939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-bundle" (OuterVolumeSpecName: "bundle") pod "bee8c9e7-c721-40d5-951c-81973bd7b143" (UID: "bee8c9e7-c721-40d5-951c-81973bd7b143"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:57:10.826383 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.826359    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee8c9e7-c721-40d5-951c-81973bd7b143-kube-api-access-7265z" (OuterVolumeSpecName: "kube-api-access-7265z") pod "bee8c9e7-c721-40d5-951c-81973bd7b143" (UID: "bee8c9e7-c721-40d5-951c-81973bd7b143"). InnerVolumeSpecName "kube-api-access-7265z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:57:10.828660 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.828640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-util" (OuterVolumeSpecName: "util") pod "bee8c9e7-c721-40d5-951c-81973bd7b143" (UID: "bee8c9e7-c721-40d5-951c-81973bd7b143"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:57:10.925065 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.925032    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-util\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:57:10.925065 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.925060    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bee8c9e7-c721-40d5-951c-81973bd7b143-bundle\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:57:10.925065 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:10.925069    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7265z\" (UniqueName: \"kubernetes.io/projected/bee8c9e7-c721-40d5-951c-81973bd7b143-kube-api-access-7265z\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:57:11.557314 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:11.557289    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq"
Sep 18 21:57:11.557314 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:11.557305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ast4cq" event={"ID":"bee8c9e7-c721-40d5-951c-81973bd7b143","Type":"ContainerDied","Data":"13745ccde3aeeb71f076cd8ce50a739b05d7d84c1b11f5f5cc4fb72eeb70b7e1"}
Sep 18 21:57:11.557506 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:11.557345    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13745ccde3aeeb71f076cd8ce50a739b05d7d84c1b11f5f5cc4fb72eeb70b7e1"
Sep 18 21:57:16.375665 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.375630    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"]
Sep 18 21:57:16.376081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.375927    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerName="extract"
Sep 18 21:57:16.376081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.375939    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerName="extract"
Sep 18 21:57:16.376081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.375962    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerName="pull"
Sep 18 21:57:16.376081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.375967    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerName="pull"
Sep 18 21:57:16.376081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.375984    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerName="util"
Sep 18 21:57:16.376081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.375989    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerName="util"
Sep 18 21:57:16.376081 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.376039    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee8c9e7-c721-40d5-951c-81973bd7b143" containerName="extract"
Sep 18 21:57:16.379384 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.379364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:16.382640 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.382621    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-78b5f\""
Sep 18 21:57:16.383166 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.383145    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 21:57:16.383275 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.383172    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 21:57:16.389386 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.389367    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 21:57:16.432776 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.432751    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"]
Sep 18 21:57:16.472165 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.472124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jjc\" (UniqueName: \"kubernetes.io/projected/08b849cf-a5b7-4831-9d0d-347e2239b1eb-kube-api-access-n7jjc\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gj275\" (UID: \"08b849cf-a5b7-4831-9d0d-347e2239b1eb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:16.472282 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.472170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/08b849cf-a5b7-4831-9d0d-347e2239b1eb-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gj275\" (UID: \"08b849cf-a5b7-4831-9d0d-347e2239b1eb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:16.572846 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.572809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jjc\" (UniqueName: \"kubernetes.io/projected/08b849cf-a5b7-4831-9d0d-347e2239b1eb-kube-api-access-n7jjc\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gj275\" (UID: \"08b849cf-a5b7-4831-9d0d-347e2239b1eb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:16.572846 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.572849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/08b849cf-a5b7-4831-9d0d-347e2239b1eb-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gj275\" (UID: \"08b849cf-a5b7-4831-9d0d-347e2239b1eb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:16.575374 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.575342    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/08b849cf-a5b7-4831-9d0d-347e2239b1eb-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gj275\" (UID: \"08b849cf-a5b7-4831-9d0d-347e2239b1eb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:16.597384 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.597356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jjc\" (UniqueName: \"kubernetes.io/projected/08b849cf-a5b7-4831-9d0d-347e2239b1eb-kube-api-access-n7jjc\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gj275\" (UID: \"08b849cf-a5b7-4831-9d0d-347e2239b1eb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:16.689536 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:16.689459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:17.061582 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:17.061500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"]
Sep 18 21:57:17.066637 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:57:17.066611    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b849cf_a5b7_4831_9d0d_347e2239b1eb.slice/crio-f8b7c2c0335ea0a3a028c9b76ecefe5bf5c39d327d50b01e56f8d1969322200f WatchSource:0}: Error finding container f8b7c2c0335ea0a3a028c9b76ecefe5bf5c39d327d50b01e56f8d1969322200f: Status 404 returned error can't find the container with id f8b7c2c0335ea0a3a028c9b76ecefe5bf5c39d327d50b01e56f8d1969322200f
Sep 18 21:57:17.582575 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:17.582538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275" event={"ID":"08b849cf-a5b7-4831-9d0d-347e2239b1eb","Type":"ContainerStarted","Data":"f8b7c2c0335ea0a3a028c9b76ecefe5bf5c39d327d50b01e56f8d1969322200f"}
Sep 18 21:57:20.600355 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.600318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275" event={"ID":"08b849cf-a5b7-4831-9d0d-347e2239b1eb","Type":"ContainerStarted","Data":"91efa9855cda08813b8dd989bdd96144746e20e46271fed781575cfe3b5576c1"}
Sep 18 21:57:20.600746 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.600394    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:20.628713 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.628664    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275" podStartSLOduration=1.769615962 podStartE2EDuration="4.628644615s" totalImagesPullingTime="2.859028653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:57:20.625890613 +0000 UTC m=+397.659069520" watchObservedRunningTime="2026-09-18 21:57:20.628644615 +0000 UTC m=+397.661823509"
Sep 18 21:57:20.798878 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.798848    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-kl4cx"]
Sep 18 21:57:20.802177 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.802160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:20.805119 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.805095    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-c4mjm\""
Sep 18 21:57:20.805381 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.805367    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 21:57:20.805623 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.805610    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 21:57:20.817332 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.817313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-kl4cx"]
Sep 18 21:57:20.906911 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.906878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rshp\" (UniqueName: \"kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-kube-api-access-4rshp\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:20.906911 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.906914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-certificates\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:20.907135 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:20.906934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-cabundle0\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.007908 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.007879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rshp\" (UniqueName: \"kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-kube-api-access-4rshp\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.008090 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.007916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-certificates\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.008090 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.007932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-cabundle0\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.008090 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:57:21.008074    2582 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 18 21:57:21.008213 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:57:21.008098    2582 secret.go:281] references non-existent secret key: ca.crt
Sep 18 21:57:21.008213 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:57:21.008111    2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 21:57:21.008213 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:57:21.008128    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-kl4cx: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 18 21:57:21.008213 ip-10-0-142-209 kubenswrapper[2582]: E0918 21:57:21.008208    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-certificates podName:b42b0da7-4cbd-4d6b-91bf-fb060a22192b nodeName:}" failed. No retries permitted until 2026-09-18 21:57:21.508186287 +0000 UTC m=+398.541365163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-certificates") pod "keda-operator-579c8958f7-kl4cx" (UID: "b42b0da7-4cbd-4d6b-91bf-fb060a22192b") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 18 21:57:21.008721 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.008700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-cabundle0\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.024900 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.024871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rshp\" (UniqueName: \"kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-kube-api-access-4rshp\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.308897 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.308814    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"]
Sep 18 21:57:21.312297 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.312279    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.315283 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.315255    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 21:57:21.339884 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.339851    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"]
Sep 18 21:57:21.411830 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.411798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8136d057-d974-4d61-86ee-674c8bd70a67-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.411991 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.411839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8136d057-d974-4d61-86ee-674c8bd70a67-certificates\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.411991 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.411869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspmm\" (UniqueName: \"kubernetes.io/projected/8136d057-d974-4d61-86ee-674c8bd70a67-kube-api-access-jspmm\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.513284 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.513254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-certificates\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.513457 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.513295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8136d057-d974-4d61-86ee-674c8bd70a67-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.513457 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.513314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8136d057-d974-4d61-86ee-674c8bd70a67-certificates\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.513588 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.513564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspmm\" (UniqueName: \"kubernetes.io/projected/8136d057-d974-4d61-86ee-674c8bd70a67-kube-api-access-jspmm\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.513742 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.513717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8136d057-d974-4d61-86ee-674c8bd70a67-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.515788 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.515764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b42b0da7-4cbd-4d6b-91bf-fb060a22192b-certificates\") pod \"keda-operator-579c8958f7-kl4cx\" (UID: \"b42b0da7-4cbd-4d6b-91bf-fb060a22192b\") " pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.515936 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.515890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8136d057-d974-4d61-86ee-674c8bd70a67-certificates\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.535272 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.535250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspmm\" (UniqueName: \"kubernetes.io/projected/8136d057-d974-4d61-86ee-674c8bd70a67-kube-api-access-jspmm\") pod \"keda-metrics-apiserver-77857b7b98-66mqh\" (UID: \"8136d057-d974-4d61-86ee-674c8bd70a67\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.622888 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.622863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:21.651097 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.651067    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-2p69k"]
Sep 18 21:57:21.654484 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.654463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:21.660090 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.659926    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 21:57:21.712329 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.712302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:21.751024 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.750987    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-2p69k"]
Sep 18 21:57:21.816278 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.816244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c229ffe7-ff33-42cf-a114-0a55ada9c4c7-certificates\") pod \"keda-admission-6c4d864b86-2p69k\" (UID: \"c229ffe7-ff33-42cf-a114-0a55ada9c4c7\") " pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:21.816413 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.816300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwpf\" (UniqueName: \"kubernetes.io/projected/c229ffe7-ff33-42cf-a114-0a55ada9c4c7-kube-api-access-dhwpf\") pod \"keda-admission-6c4d864b86-2p69k\" (UID: \"c229ffe7-ff33-42cf-a114-0a55ada9c4c7\") " pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:21.917752 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.917670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c229ffe7-ff33-42cf-a114-0a55ada9c4c7-certificates\") pod \"keda-admission-6c4d864b86-2p69k\" (UID: \"c229ffe7-ff33-42cf-a114-0a55ada9c4c7\") " pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:21.917752 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.917723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwpf\" (UniqueName: \"kubernetes.io/projected/c229ffe7-ff33-42cf-a114-0a55ada9c4c7-kube-api-access-dhwpf\") pod \"keda-admission-6c4d864b86-2p69k\" (UID: \"c229ffe7-ff33-42cf-a114-0a55ada9c4c7\") " pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:21.920220 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.920199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c229ffe7-ff33-42cf-a114-0a55ada9c4c7-certificates\") pod \"keda-admission-6c4d864b86-2p69k\" (UID: \"c229ffe7-ff33-42cf-a114-0a55ada9c4c7\") " pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:21.931462 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.931432    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"]
Sep 18 21:57:21.936563 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:57:21.936536    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8136d057_d974_4d61_86ee_674c8bd70a67.slice/crio-615580c4f931fa2c61e37e0edefb0286cc10127d464e1debb2f31f03a35f6379 WatchSource:0}: Error finding container 615580c4f931fa2c61e37e0edefb0286cc10127d464e1debb2f31f03a35f6379: Status 404 returned error can't find the container with id 615580c4f931fa2c61e37e0edefb0286cc10127d464e1debb2f31f03a35f6379
Sep 18 21:57:21.937659 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.937641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwpf\" (UniqueName: \"kubernetes.io/projected/c229ffe7-ff33-42cf-a114-0a55ada9c4c7-kube-api-access-dhwpf\") pod \"keda-admission-6c4d864b86-2p69k\" (UID: \"c229ffe7-ff33-42cf-a114-0a55ada9c4c7\") " pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:21.965844 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:21.965797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:22.014334 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:22.014303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-kl4cx"]
Sep 18 21:57:22.259077 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:22.258991    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-2p69k"]
Sep 18 21:57:22.262777 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:57:22.262748    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc229ffe7_ff33_42cf_a114_0a55ada9c4c7.slice/crio-9a371ab10ec0001569011a63fac2c6ef29f9899b1ae4c73f9b91f0030a30f636 WatchSource:0}: Error finding container 9a371ab10ec0001569011a63fac2c6ef29f9899b1ae4c73f9b91f0030a30f636: Status 404 returned error can't find the container with id 9a371ab10ec0001569011a63fac2c6ef29f9899b1ae4c73f9b91f0030a30f636
Sep 18 21:57:22.609469 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:22.609371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-2p69k" event={"ID":"c229ffe7-ff33-42cf-a114-0a55ada9c4c7","Type":"ContainerStarted","Data":"9a371ab10ec0001569011a63fac2c6ef29f9899b1ae4c73f9b91f0030a30f636"}
Sep 18 21:57:22.610472 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:22.610446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh" event={"ID":"8136d057-d974-4d61-86ee-674c8bd70a67","Type":"ContainerStarted","Data":"615580c4f931fa2c61e37e0edefb0286cc10127d464e1debb2f31f03a35f6379"}
Sep 18 21:57:22.611534 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:22.611512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-kl4cx" event={"ID":"b42b0da7-4cbd-4d6b-91bf-fb060a22192b","Type":"ContainerStarted","Data":"c2af668826578559de280ac78f113769b157d6f3b8dfaa63d6478d8981707621"}
Sep 18 21:57:26.633914 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.633872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh" event={"ID":"8136d057-d974-4d61-86ee-674c8bd70a67","Type":"ContainerStarted","Data":"fe16a4e0bf51e416376bf6638e53eda3c9b1682b38c4ee07b827e3741b35ef38"}
Sep 18 21:57:26.634363 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.633992    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:26.635561 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.635535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-kl4cx" event={"ID":"b42b0da7-4cbd-4d6b-91bf-fb060a22192b","Type":"ContainerStarted","Data":"de3c5931bea8c65938bc84c6c6bd8c5f592741b450dc3ae699462aec7c859d62"}
Sep 18 21:57:26.635698 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.635635    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:26.637011 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.636989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-2p69k" event={"ID":"c229ffe7-ff33-42cf-a114-0a55ada9c4c7","Type":"ContainerStarted","Data":"ee834d0e88bd46d9c5fc5e48651925079e784e7f01ad405acf0620b74d8aed37"}
Sep 18 21:57:26.637119 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.637108    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:57:26.656494 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.656454    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh" podStartSLOduration=1.569282608 podStartE2EDuration="5.656439141s" totalImagesPullingTime="4.087156533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:57:26.654075692 +0000 UTC m=+403.687254587" watchObservedRunningTime="2026-09-18 21:57:26.656439141 +0000 UTC m=+403.689618036"
Sep 18 21:57:26.678047 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.678005    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-2p69k" podStartSLOduration=1.9203368840000001 podStartE2EDuration="5.677986383s" totalImagesPullingTime="3.757649499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:57:26.676107768 +0000 UTC m=+403.709286665" watchObservedRunningTime="2026-09-18 21:57:26.677986383 +0000 UTC m=+403.711165278"
Sep 18 21:57:26.702558 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:26.702502    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-kl4cx" podStartSLOduration=2.688404424 podStartE2EDuration="6.702486612s" totalImagesPullingTime="4.014082188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:57:26.700575216 +0000 UTC m=+403.733754111" watchObservedRunningTime="2026-09-18 21:57:26.702486612 +0000 UTC m=+403.735665507"
Sep 18 21:57:37.646406 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:37.646378    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-66mqh"
Sep 18 21:57:41.606702 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:41.606663    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 21:57:47.643434 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:47.643397    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-kl4cx"
Sep 18 21:57:47.643854 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:57:47.643461    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-2p69k"
Sep 18 21:58:11.715326 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.715251    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"]
Sep 18 21:58:11.718730 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.718712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.721148 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.721125    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 21:58:11.721293 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.721276    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 21:58:11.721836 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.721818    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ggxp8\""
Sep 18 21:58:11.728786 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.728765    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"]
Sep 18 21:58:11.831322 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.831290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.831484 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.831336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztm4\" (UniqueName: \"kubernetes.io/projected/6a90a531-41fa-4a5c-8969-b5687c9569fb-kube-api-access-fztm4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.831484 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.831367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.932582 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.932550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztm4\" (UniqueName: \"kubernetes.io/projected/6a90a531-41fa-4a5c-8969-b5687c9569fb-kube-api-access-fztm4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.932769 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.932590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.932769 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.932633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.932937 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.932921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.933029 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.932979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:11.940891 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:11.940871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztm4\" (UniqueName: \"kubernetes.io/projected/6a90a531-41fa-4a5c-8969-b5687c9569fb-kube-api-access-fztm4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:12.028993 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:12.028891    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:12.303506 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:12.303342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"]
Sep 18 21:58:12.306133 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:58:12.306102    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a90a531_41fa_4a5c_8969_b5687c9569fb.slice/crio-4d6320d48bcaa6f81c7580e52923fa38c3b2bf154f3ff42e6104c009783820f2 WatchSource:0}: Error finding container 4d6320d48bcaa6f81c7580e52923fa38c3b2bf154f3ff42e6104c009783820f2: Status 404 returned error can't find the container with id 4d6320d48bcaa6f81c7580e52923fa38c3b2bf154f3ff42e6104c009783820f2
Sep 18 21:58:12.849609 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:12.849578    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerID="4e1e42be2ffbabd12bbbdaa5ceb4aeaba7b359adda1c4a9776277527ce854dd4" exitCode=0
Sep 18 21:58:12.850085 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:12.849662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78" event={"ID":"6a90a531-41fa-4a5c-8969-b5687c9569fb","Type":"ContainerDied","Data":"4e1e42be2ffbabd12bbbdaa5ceb4aeaba7b359adda1c4a9776277527ce854dd4"}
Sep 18 21:58:12.850085 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:12.849713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78" event={"ID":"6a90a531-41fa-4a5c-8969-b5687c9569fb","Type":"ContainerStarted","Data":"4d6320d48bcaa6f81c7580e52923fa38c3b2bf154f3ff42e6104c009783820f2"}
Sep 18 21:58:14.860911 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:14.860883    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerID="42c558af82c75eed29f20cda8e288f780eca317860d9d1e274dcb6ac86e36887" exitCode=0
Sep 18 21:58:14.861253 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:14.860934    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78" event={"ID":"6a90a531-41fa-4a5c-8969-b5687c9569fb","Type":"ContainerDied","Data":"42c558af82c75eed29f20cda8e288f780eca317860d9d1e274dcb6ac86e36887"}
Sep 18 21:58:15.867966 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:15.867897    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerID="1d8c5a1cb2527eeb2af9fca4ee3ee80dd7c20a8a91f26127f9b2ad4dfe84f025" exitCode=0
Sep 18 21:58:15.868327 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:15.867983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78" event={"ID":"6a90a531-41fa-4a5c-8969-b5687c9569fb","Type":"ContainerDied","Data":"1d8c5a1cb2527eeb2af9fca4ee3ee80dd7c20a8a91f26127f9b2ad4dfe84f025"}
Sep 18 21:58:17.009904 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.009881    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:17.076111 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.076079    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fztm4\" (UniqueName: \"kubernetes.io/projected/6a90a531-41fa-4a5c-8969-b5687c9569fb-kube-api-access-fztm4\") pod \"6a90a531-41fa-4a5c-8969-b5687c9569fb\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") "
Sep 18 21:58:17.076111 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.076114    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-bundle\") pod \"6a90a531-41fa-4a5c-8969-b5687c9569fb\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") "
Sep 18 21:58:17.076336 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.076138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-util\") pod \"6a90a531-41fa-4a5c-8969-b5687c9569fb\" (UID: \"6a90a531-41fa-4a5c-8969-b5687c9569fb\") "
Sep 18 21:58:17.076826 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.076801    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-bundle" (OuterVolumeSpecName: "bundle") pod "6a90a531-41fa-4a5c-8969-b5687c9569fb" (UID: "6a90a531-41fa-4a5c-8969-b5687c9569fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:58:17.078214 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.078185    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a90a531-41fa-4a5c-8969-b5687c9569fb-kube-api-access-fztm4" (OuterVolumeSpecName: "kube-api-access-fztm4") pod "6a90a531-41fa-4a5c-8969-b5687c9569fb" (UID: "6a90a531-41fa-4a5c-8969-b5687c9569fb"). InnerVolumeSpecName "kube-api-access-fztm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:58:17.082267 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.082231    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-util" (OuterVolumeSpecName: "util") pod "6a90a531-41fa-4a5c-8969-b5687c9569fb" (UID: "6a90a531-41fa-4a5c-8969-b5687c9569fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:58:17.177428 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.177355    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-util\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:58:17.177428 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.177381    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fztm4\" (UniqueName: \"kubernetes.io/projected/6a90a531-41fa-4a5c-8969-b5687c9569fb-kube-api-access-fztm4\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:58:17.177428 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.177392    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a90a531-41fa-4a5c-8969-b5687c9569fb-bundle\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:58:17.878869 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.878835    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78" event={"ID":"6a90a531-41fa-4a5c-8969-b5687c9569fb","Type":"ContainerDied","Data":"4d6320d48bcaa6f81c7580e52923fa38c3b2bf154f3ff42e6104c009783820f2"}
Sep 18 21:58:17.879056 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.878881    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6320d48bcaa6f81c7580e52923fa38c3b2bf154f3ff42e6104c009783820f2"
Sep 18 21:58:17.879056 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:17.878884    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540v8p78"
Sep 18 21:58:34.114131 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114097    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"]
Sep 18 21:58:34.114496 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114421    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerName="util"
Sep 18 21:58:34.114496 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114431    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerName="util"
Sep 18 21:58:34.114496 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114439    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerName="extract"
Sep 18 21:58:34.114496 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114445    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerName="extract"
Sep 18 21:58:34.114496 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114459    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerName="pull"
Sep 18 21:58:34.114496 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114464    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerName="pull"
Sep 18 21:58:34.114681 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.114519    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a90a531-41fa-4a5c-8969-b5687c9569fb" containerName="extract"
Sep 18 21:58:34.117487 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.117471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.121728 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.121709    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 21:58:34.122289 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.122271    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 21:58:34.122490 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.122475    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ggxp8\""
Sep 18 21:58:34.157855 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.157827    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"]
Sep 18 21:58:34.208242 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.208213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.208380 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.208267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.208380 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.208291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrlsl\" (UniqueName: \"kubernetes.io/projected/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-kube-api-access-lrlsl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.309015 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.308981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.309209 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.309031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.309209 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.309051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrlsl\" (UniqueName: \"kubernetes.io/projected/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-kube-api-access-lrlsl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.309451 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.309427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.309516 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.309488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.334528 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.334508    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrlsl\" (UniqueName: \"kubernetes.io/projected/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-kube-api-access-lrlsl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.426677 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.426641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:34.745086 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.744862    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"]
Sep 18 21:58:34.747654 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:58:34.747626    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648ec5b9_8f70_4626_8b2c_2b6f2aea23d7.slice/crio-4dae743c418fbf1c34248f63c88f0c5af5dccd720774b1a527d550cc6cb5a7a3 WatchSource:0}: Error finding container 4dae743c418fbf1c34248f63c88f0c5af5dccd720774b1a527d550cc6cb5a7a3: Status 404 returned error can't find the container with id 4dae743c418fbf1c34248f63c88f0c5af5dccd720774b1a527d550cc6cb5a7a3
Sep 18 21:58:34.958244 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.958209    2582 generic.go:356] "Generic (PLEG): container finished" podID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerID="813c9548be9446aa8ce0efd173d4ab25ce2f1952544460eb91a5332c5315faf3" exitCode=0
Sep 18 21:58:34.958415 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.958293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k" event={"ID":"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7","Type":"ContainerDied","Data":"813c9548be9446aa8ce0efd173d4ab25ce2f1952544460eb91a5332c5315faf3"}
Sep 18 21:58:34.958415 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:34.958349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k" event={"ID":"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7","Type":"ContainerStarted","Data":"4dae743c418fbf1c34248f63c88f0c5af5dccd720774b1a527d550cc6cb5a7a3"}
Sep 18 21:58:37.977659 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:37.977624    2582 generic.go:356] "Generic (PLEG): container finished" podID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerID="294cad207e509618347a910cd1a6c6c647ec564ba415ea04943a4a19f206ebdf" exitCode=0
Sep 18 21:58:37.978043 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:37.977720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k" event={"ID":"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7","Type":"ContainerDied","Data":"294cad207e509618347a910cd1a6c6c647ec564ba415ea04943a4a19f206ebdf"}
Sep 18 21:58:38.984390 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:38.984357    2582 generic.go:356] "Generic (PLEG): container finished" podID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerID="3e9f2dbb0fb3b4c4e994492c28040519d44297e2c67907afbb29e316aff8d514" exitCode=0
Sep 18 21:58:38.984765 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:38.984432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k" event={"ID":"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7","Type":"ContainerDied","Data":"3e9f2dbb0fb3b4c4e994492c28040519d44297e2c67907afbb29e316aff8d514"}
Sep 18 21:58:40.114814 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.114790    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:40.157930 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.157901    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrlsl\" (UniqueName: \"kubernetes.io/projected/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-kube-api-access-lrlsl\") pod \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") "
Sep 18 21:58:40.158110 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.158003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-util\") pod \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") "
Sep 18 21:58:40.158110 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.158031    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-bundle\") pod \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\" (UID: \"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7\") "
Sep 18 21:58:40.158476 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.158442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-bundle" (OuterVolumeSpecName: "bundle") pod "648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" (UID: "648ec5b9-8f70-4626-8b2c-2b6f2aea23d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:58:40.160323 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.160293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-kube-api-access-lrlsl" (OuterVolumeSpecName: "kube-api-access-lrlsl") pod "648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" (UID: "648ec5b9-8f70-4626-8b2c-2b6f2aea23d7"). InnerVolumeSpecName "kube-api-access-lrlsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:58:40.164205 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.164180    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-util" (OuterVolumeSpecName: "util") pod "648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" (UID: "648ec5b9-8f70-4626-8b2c-2b6f2aea23d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:58:40.259144 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.259071    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrlsl\" (UniqueName: \"kubernetes.io/projected/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-kube-api-access-lrlsl\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:58:40.259144 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.259102    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-util\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:58:40.259144 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.259114    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/648ec5b9-8f70-4626-8b2c-2b6f2aea23d7-bundle\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:58:40.995725 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.995695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k" event={"ID":"648ec5b9-8f70-4626-8b2c-2b6f2aea23d7","Type":"ContainerDied","Data":"4dae743c418fbf1c34248f63c88f0c5af5dccd720774b1a527d550cc6cb5a7a3"}
Sep 18 21:58:40.995725 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.995724    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwht2k"
Sep 18 21:58:40.995907 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:58:40.995738    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dae743c418fbf1c34248f63c88f0c5af5dccd720774b1a527d550cc6cb5a7a3"
Sep 18 21:59:05.174148 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174111    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"]
Sep 18 21:59:05.174744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174581    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerName="extract"
Sep 18 21:59:05.174744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174599    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerName="extract"
Sep 18 21:59:05.174744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174626    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerName="util"
Sep 18 21:59:05.174744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174634    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerName="util"
Sep 18 21:59:05.174744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174661    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerName="pull"
Sep 18 21:59:05.174744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174669    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerName="pull"
Sep 18 21:59:05.174744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.174744    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="648ec5b9-8f70-4626-8b2c-2b6f2aea23d7" containerName="extract"
Sep 18 21:59:05.177820 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.177800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.184761 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.184731    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-b6kjd\""
Sep 18 21:59:05.184865 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.184731    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 21:59:05.184865 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.184800    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 21:59:05.184865 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.184811    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 21:59:05.184865 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.184840    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 21:59:05.185153 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.185138    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 21:59:05.210187 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.210165    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"]
Sep 18 21:59:05.262870 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.262838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb800afb-8362-493c-a4c4-ba434637a7bd-manager-config\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.262870 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.262870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb800afb-8362-493c-a4c4-ba434637a7bd-cert\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.263060 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.262893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb800afb-8362-493c-a4c4-ba434637a7bd-metrics-cert\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.263060 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.262908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnpss\" (UniqueName: \"kubernetes.io/projected/bb800afb-8362-493c-a4c4-ba434637a7bd-kube-api-access-gnpss\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.363635 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.363598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb800afb-8362-493c-a4c4-ba434637a7bd-manager-config\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.363635 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.363631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb800afb-8362-493c-a4c4-ba434637a7bd-cert\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.363860 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.363654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb800afb-8362-493c-a4c4-ba434637a7bd-metrics-cert\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.363860 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.363674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnpss\" (UniqueName: \"kubernetes.io/projected/bb800afb-8362-493c-a4c4-ba434637a7bd-kube-api-access-gnpss\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.364323 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.364298    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb800afb-8362-493c-a4c4-ba434637a7bd-manager-config\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.366243 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.366215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb800afb-8362-493c-a4c4-ba434637a7bd-metrics-cert\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.366341 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.366257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb800afb-8362-493c-a4c4-ba434637a7bd-cert\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.382450 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.382426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnpss\" (UniqueName: \"kubernetes.io/projected/bb800afb-8362-493c-a4c4-ba434637a7bd-kube-api-access-gnpss\") pod \"lws-controller-manager-7fdf8f455-crjpm\" (UID: \"bb800afb-8362-493c-a4c4-ba434637a7bd\") " pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.486930 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.486859    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:05.764452 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:05.764381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"]
Sep 18 21:59:05.767622 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:59:05.767592    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb800afb_8362_493c_a4c4_ba434637a7bd.slice/crio-58e518b494287be838333b6706a4cad6a6a101f7e8e6146cc7bec94781e85c76 WatchSource:0}: Error finding container 58e518b494287be838333b6706a4cad6a6a101f7e8e6146cc7bec94781e85c76: Status 404 returned error can't find the container with id 58e518b494287be838333b6706a4cad6a6a101f7e8e6146cc7bec94781e85c76
Sep 18 21:59:06.107592 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:06.107515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm" event={"ID":"bb800afb-8362-493c-a4c4-ba434637a7bd","Type":"ContainerStarted","Data":"58e518b494287be838333b6706a4cad6a6a101f7e8e6146cc7bec94781e85c76"}
Sep 18 21:59:09.124427 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:09.124390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm" event={"ID":"bb800afb-8362-493c-a4c4-ba434637a7bd","Type":"ContainerStarted","Data":"21d11fdb77afabed8cbbc2bf33cd1d762fb93e9c2a3f33d8b0691c08f2100f63"}
Sep 18 21:59:09.124826 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:09.124531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:09.161505 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:09.161451    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm" podStartSLOduration=1.7005283709999999 podStartE2EDuration="4.161409942s" totalImagesPullingTime="2.460881571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 21:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 21:59:09.161212374 +0000 UTC m=+506.194391281" watchObservedRunningTime="2026-09-18 21:59:09.161409942 +0000 UTC m=+506.194588839"
Sep 18 21:59:20.131009 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:20.130976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7fdf8f455-crjpm"
Sep 18 21:59:24.771472 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.771438    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"]
Sep 18 21:59:24.778837 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.778815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.782375 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.782355    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 21:59:24.783054 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.783035    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ggxp8\""
Sep 18 21:59:24.783127 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.783090    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 21:59:24.795629 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.795602    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"]
Sep 18 21:59:24.805065 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.805042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.805147 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.805084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.805187 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.805152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4h9x\" (UniqueName: \"kubernetes.io/projected/91b171c6-2841-49c2-a81b-33df9b575b99-kube-api-access-f4h9x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.905767 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.905719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.905940 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.905774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.905940 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.905910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4h9x\" (UniqueName: \"kubernetes.io/projected/91b171c6-2841-49c2-a81b-33df9b575b99-kube-api-access-f4h9x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.906144 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.906124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.906219 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.906193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.913316 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.913291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"]
Sep 18 21:59:24.917249 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.917222    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:24.917447 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.917427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4h9x\" (UniqueName: \"kubernetes.io/projected/91b171c6-2841-49c2-a81b-33df9b575b99-kube-api-access-f4h9x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:24.933744 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:24.933720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"]
Sep 18 21:59:25.006394 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.006364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46t4t\" (UniqueName: \"kubernetes.io/projected/50ced5a9-8dee-4107-8ad9-f3df17912b2c-kube-api-access-46t4t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.006543 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.006415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.006543 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.006441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.039877 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.039790    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"]
Sep 18 21:59:25.043664 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.043644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.055468 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.055442    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"]
Sep 18 21:59:25.087702 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.087678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:25.107014 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.106990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.107126 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.107061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkvpc\" (UniqueName: \"kubernetes.io/projected/baf45629-8364-49a1-ace4-6d474ea305ce-kube-api-access-pkvpc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.107126 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.107094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.107238 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.107129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46t4t\" (UniqueName: \"kubernetes.io/projected/50ced5a9-8dee-4107-8ad9-f3df17912b2c-kube-api-access-46t4t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.107458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.107225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.107458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.107273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.107458 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.107352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.107677 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.107547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.121768 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.121734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46t4t\" (UniqueName: \"kubernetes.io/projected/50ced5a9-8dee-4107-8ad9-f3df17912b2c-kube-api-access-46t4t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.130754 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.130731    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"]
Sep 18 21:59:25.135808 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.135788    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.157427 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.157356    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"]
Sep 18 21:59:25.207799 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.207773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.207969 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.207840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.207969 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.207862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkvpc\" (UniqueName: \"kubernetes.io/projected/baf45629-8364-49a1-ace4-6d474ea305ce-kube-api-access-pkvpc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.207969 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.207883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.207969 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.207906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.207969 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.207936    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxh2c\" (UniqueName: \"kubernetes.io/projected/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-kube-api-access-kxh2c\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.208236 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.208155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.208236 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.208196    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.218410 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.218389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkvpc\" (UniqueName: \"kubernetes.io/projected/baf45629-8364-49a1-ace4-6d474ea305ce-kube-api-access-pkvpc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.232117 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.232099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:25.308924 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.308860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.308924 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.308897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxh2c\" (UniqueName: \"kubernetes.io/projected/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-kube-api-access-kxh2c\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.309095 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.309008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.309314 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.309292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.309354 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.309325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.321999 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.321978    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxh2c\" (UniqueName: \"kubernetes.io/projected/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-kube-api-access-kxh2c\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.352771 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.352744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:25.375029 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.375005    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"]
Sep 18 21:59:25.376552 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:59:25.376527    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b171c6_2841_49c2_a81b_33df9b575b99.slice/crio-5bd6e0e5c62c450b5c36161c5c6a38571b33a4f89b09cf594b36d3f08d8ced38 WatchSource:0}: Error finding container 5bd6e0e5c62c450b5c36161c5c6a38571b33a4f89b09cf594b36d3f08d8ced38: Status 404 returned error can't find the container with id 5bd6e0e5c62c450b5c36161c5c6a38571b33a4f89b09cf594b36d3f08d8ced38
Sep 18 21:59:25.449716 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.449685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:25.510661 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.510632    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"]
Sep 18 21:59:25.529935 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:59:25.529908    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ced5a9_8dee_4107_8ad9_f3df17912b2c.slice/crio-cca37d8515ebf0199d511b6892e44cd2e9d78a4181e405e8081ed3facac3bb26 WatchSource:0}: Error finding container cca37d8515ebf0199d511b6892e44cd2e9d78a4181e405e8081ed3facac3bb26: Status 404 returned error can't find the container with id cca37d8515ebf0199d511b6892e44cd2e9d78a4181e405e8081ed3facac3bb26
Sep 18 21:59:25.638449 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.638423    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"]
Sep 18 21:59:25.682772 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:59:25.682743    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf45629_8364_49a1_ace4_6d474ea305ce.slice/crio-53f1211574b06bc466b24feeb612000897023c03a671ab0b6c876b970003256f WatchSource:0}: Error finding container 53f1211574b06bc466b24feeb612000897023c03a671ab0b6c876b970003256f: Status 404 returned error can't find the container with id 53f1211574b06bc466b24feeb612000897023c03a671ab0b6c876b970003256f
Sep 18 21:59:25.727702 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:25.727675    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"]
Sep 18 21:59:25.733282 ip-10-0-142-209 kubenswrapper[2582]: W0918 21:59:25.733256    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147a4f4b_5e31_4f0c_a4e7_2b4cdc78eecc.slice/crio-18b1ba5c8e2eaf99ac148a98a1ec651e825ba6d6d950812fd7418ca186054e3f WatchSource:0}: Error finding container 18b1ba5c8e2eaf99ac148a98a1ec651e825ba6d6d950812fd7418ca186054e3f: Status 404 returned error can't find the container with id 18b1ba5c8e2eaf99ac148a98a1ec651e825ba6d6d950812fd7418ca186054e3f
Sep 18 21:59:26.204192 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.204164    2582 generic.go:356] "Generic (PLEG): container finished" podID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerID="4e1fc6d2c723b6ab4af4a48ce66bc40a6c7db808418f6bb0e307b7f7ac8fb33d" exitCode=0
Sep 18 21:59:26.204606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.204235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j" event={"ID":"50ced5a9-8dee-4107-8ad9-f3df17912b2c","Type":"ContainerDied","Data":"4e1fc6d2c723b6ab4af4a48ce66bc40a6c7db808418f6bb0e307b7f7ac8fb33d"}
Sep 18 21:59:26.204606 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.204271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j" event={"ID":"50ced5a9-8dee-4107-8ad9-f3df17912b2c","Type":"ContainerStarted","Data":"cca37d8515ebf0199d511b6892e44cd2e9d78a4181e405e8081ed3facac3bb26"}
Sep 18 21:59:26.205958 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.205908    2582 generic.go:356] "Generic (PLEG): container finished" podID="91b171c6-2841-49c2-a81b-33df9b575b99" containerID="c223c0b4409d1c912ff0d86452dbffbf9990694151da4d07df28831d7251c960" exitCode=0
Sep 18 21:59:26.206069 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.206051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg" event={"ID":"91b171c6-2841-49c2-a81b-33df9b575b99","Type":"ContainerDied","Data":"c223c0b4409d1c912ff0d86452dbffbf9990694151da4d07df28831d7251c960"}
Sep 18 21:59:26.206131 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.206094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg" event={"ID":"91b171c6-2841-49c2-a81b-33df9b575b99","Type":"ContainerStarted","Data":"5bd6e0e5c62c450b5c36161c5c6a38571b33a4f89b09cf594b36d3f08d8ced38"}
Sep 18 21:59:26.207792 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.207678    2582 generic.go:356] "Generic (PLEG): container finished" podID="baf45629-8364-49a1-ace4-6d474ea305ce" containerID="78f1d2dda0e3f66f7acb11f75d2d6458c729a088cc780694e89762f7c1af5996" exitCode=0
Sep 18 21:59:26.207792 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.207747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8" event={"ID":"baf45629-8364-49a1-ace4-6d474ea305ce","Type":"ContainerDied","Data":"78f1d2dda0e3f66f7acb11f75d2d6458c729a088cc780694e89762f7c1af5996"}
Sep 18 21:59:26.207928 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.207794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8" event={"ID":"baf45629-8364-49a1-ace4-6d474ea305ce","Type":"ContainerStarted","Data":"53f1211574b06bc466b24feeb612000897023c03a671ab0b6c876b970003256f"}
Sep 18 21:59:26.209519 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.209499    2582 generic.go:356] "Generic (PLEG): container finished" podID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerID="a58fb0a9abee40bcad14631a18e1760feb808efef6f82b610980a8137206fd38" exitCode=0
Sep 18 21:59:26.209619 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.209593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn" event={"ID":"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc","Type":"ContainerDied","Data":"a58fb0a9abee40bcad14631a18e1760feb808efef6f82b610980a8137206fd38"}
Sep 18 21:59:26.209672 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:26.209644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn" event={"ID":"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc","Type":"ContainerStarted","Data":"18b1ba5c8e2eaf99ac148a98a1ec651e825ba6d6d950812fd7418ca186054e3f"}
Sep 18 21:59:27.218034 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:27.218002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j" event={"ID":"50ced5a9-8dee-4107-8ad9-f3df17912b2c","Type":"ContainerStarted","Data":"3fbd885e13c461c558ce976d59152f6586958408585752ba78d52f4fca5e540c"}
Sep 18 21:59:27.219878 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:27.219852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg" event={"ID":"91b171c6-2841-49c2-a81b-33df9b575b99","Type":"ContainerStarted","Data":"13b1635c9beba3418afd936674b6060fd520e6656628d91fa9d9c3a92631c57f"}
Sep 18 21:59:28.227267 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.227237    2582 generic.go:356] "Generic (PLEG): container finished" podID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerID="3fbd885e13c461c558ce976d59152f6586958408585752ba78d52f4fca5e540c" exitCode=0
Sep 18 21:59:28.227711 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.227329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j" event={"ID":"50ced5a9-8dee-4107-8ad9-f3df17912b2c","Type":"ContainerDied","Data":"3fbd885e13c461c558ce976d59152f6586958408585752ba78d52f4fca5e540c"}
Sep 18 21:59:28.230578 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.230548    2582 generic.go:356] "Generic (PLEG): container finished" podID="91b171c6-2841-49c2-a81b-33df9b575b99" containerID="13b1635c9beba3418afd936674b6060fd520e6656628d91fa9d9c3a92631c57f" exitCode=0
Sep 18 21:59:28.230689 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.230662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg" event={"ID":"91b171c6-2841-49c2-a81b-33df9b575b99","Type":"ContainerDied","Data":"13b1635c9beba3418afd936674b6060fd520e6656628d91fa9d9c3a92631c57f"}
Sep 18 21:59:28.234255 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.234232    2582 generic.go:356] "Generic (PLEG): container finished" podID="baf45629-8364-49a1-ace4-6d474ea305ce" containerID="df90e6d6de8843176ca5bbd5b46ad4e50aa03b2f452fd51d96838921d76c740e" exitCode=0
Sep 18 21:59:28.234356 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.234313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8" event={"ID":"baf45629-8364-49a1-ace4-6d474ea305ce","Type":"ContainerDied","Data":"df90e6d6de8843176ca5bbd5b46ad4e50aa03b2f452fd51d96838921d76c740e"}
Sep 18 21:59:28.236179 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.236158    2582 generic.go:356] "Generic (PLEG): container finished" podID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerID="61499aea5ae44751918b9dc605257bc63c4cbe07abeb3665172be0c686b0fcec" exitCode=0
Sep 18 21:59:28.236293 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:28.236237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn" event={"ID":"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc","Type":"ContainerDied","Data":"61499aea5ae44751918b9dc605257bc63c4cbe07abeb3665172be0c686b0fcec"}
Sep 18 21:59:29.243717 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.243685    2582 generic.go:356] "Generic (PLEG): container finished" podID="baf45629-8364-49a1-ace4-6d474ea305ce" containerID="55d785a5052762b9113b59bc355daca552df0698196c5018817cca1052b7878c" exitCode=0
Sep 18 21:59:29.244135 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.243771    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8" event={"ID":"baf45629-8364-49a1-ace4-6d474ea305ce","Type":"ContainerDied","Data":"55d785a5052762b9113b59bc355daca552df0698196c5018817cca1052b7878c"}
Sep 18 21:59:29.245684 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.245660    2582 generic.go:356] "Generic (PLEG): container finished" podID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerID="34d8605d0fa8ba4c28bee41e49cc7f9cb77845dccae420c000bf21c349342e0a" exitCode=0
Sep 18 21:59:29.245787 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.245697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn" event={"ID":"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc","Type":"ContainerDied","Data":"34d8605d0fa8ba4c28bee41e49cc7f9cb77845dccae420c000bf21c349342e0a"}
Sep 18 21:59:29.247617 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.247598    2582 generic.go:356] "Generic (PLEG): container finished" podID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerID="4d3ff1ea3b1ef3650ee115fd65e68a911cfd73ccbc5d278cbfd8ff600e647570" exitCode=0
Sep 18 21:59:29.247696 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.247664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j" event={"ID":"50ced5a9-8dee-4107-8ad9-f3df17912b2c","Type":"ContainerDied","Data":"4d3ff1ea3b1ef3650ee115fd65e68a911cfd73ccbc5d278cbfd8ff600e647570"}
Sep 18 21:59:29.249707 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.249688    2582 generic.go:356] "Generic (PLEG): container finished" podID="91b171c6-2841-49c2-a81b-33df9b575b99" containerID="1df320cdc4aee558c236e8a73b4cff636f958d71ce06985f5d9ce9898f794966" exitCode=0
Sep 18 21:59:29.249799 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:29.249715    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg" event={"ID":"91b171c6-2841-49c2-a81b-33df9b575b99","Type":"ContainerDied","Data":"1df320cdc4aee558c236e8a73b4cff636f958d71ce06985f5d9ce9898f794966"}
Sep 18 21:59:30.390875 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.390848    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 21:59:30.439963 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.439919    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:30.451435 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.451404    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-bundle\") pod \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") "
Sep 18 21:59:30.451551 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.451466    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-util\") pod \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") "
Sep 18 21:59:30.451551 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.451507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46t4t\" (UniqueName: \"kubernetes.io/projected/50ced5a9-8dee-4107-8ad9-f3df17912b2c-kube-api-access-46t4t\") pod \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") "
Sep 18 21:59:30.451551 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.451547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-util\") pod \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") "
Sep 18 21:59:30.451688 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.451639    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-bundle\") pod \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\" (UID: \"50ced5a9-8dee-4107-8ad9-f3df17912b2c\") "
Sep 18 21:59:30.451688 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.451671    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxh2c\" (UniqueName: \"kubernetes.io/projected/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-kube-api-access-kxh2c\") pod \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\" (UID: \"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc\") "
Sep 18 21:59:30.452739 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.452707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-bundle" (OuterVolumeSpecName: "bundle") pod "50ced5a9-8dee-4107-8ad9-f3df17912b2c" (UID: "50ced5a9-8dee-4107-8ad9-f3df17912b2c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.452885 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.452714    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-bundle" (OuterVolumeSpecName: "bundle") pod "147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" (UID: "147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.454934 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.454903    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-kube-api-access-kxh2c" (OuterVolumeSpecName: "kube-api-access-kxh2c") pod "147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" (UID: "147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc"). InnerVolumeSpecName "kube-api-access-kxh2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:59:30.455723 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.455694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ced5a9-8dee-4107-8ad9-f3df17912b2c-kube-api-access-46t4t" (OuterVolumeSpecName: "kube-api-access-46t4t") pod "50ced5a9-8dee-4107-8ad9-f3df17912b2c" (UID: "50ced5a9-8dee-4107-8ad9-f3df17912b2c"). InnerVolumeSpecName "kube-api-access-46t4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:59:30.458084 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.458058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-util" (OuterVolumeSpecName: "util") pod "147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" (UID: "147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.459380 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.459354    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-util" (OuterVolumeSpecName: "util") pod "50ced5a9-8dee-4107-8ad9-f3df17912b2c" (UID: "50ced5a9-8dee-4107-8ad9-f3df17912b2c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.461418 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.461399    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:30.463498 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.463483    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:30.552300 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-bundle\") pod \"baf45629-8364-49a1-ace4-6d474ea305ce\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") "
Sep 18 21:59:30.552300 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkvpc\" (UniqueName: \"kubernetes.io/projected/baf45629-8364-49a1-ace4-6d474ea305ce-kube-api-access-pkvpc\") pod \"baf45629-8364-49a1-ace4-6d474ea305ce\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") "
Sep 18 21:59:30.552300 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552292    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4h9x\" (UniqueName: \"kubernetes.io/projected/91b171c6-2841-49c2-a81b-33df9b575b99-kube-api-access-f4h9x\") pod \"91b171c6-2841-49c2-a81b-33df9b575b99\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") "
Sep 18 21:59:30.552574 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552329    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-util\") pod \"baf45629-8364-49a1-ace4-6d474ea305ce\" (UID: \"baf45629-8364-49a1-ace4-6d474ea305ce\") "
Sep 18 21:59:30.552574 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552362    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-util\") pod \"91b171c6-2841-49c2-a81b-33df9b575b99\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") "
Sep 18 21:59:30.552574 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-bundle\") pod \"91b171c6-2841-49c2-a81b-33df9b575b99\" (UID: \"91b171c6-2841-49c2-a81b-33df9b575b99\") "
Sep 18 21:59:30.552732 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552625    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-util\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.552732 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552642    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50ced5a9-8dee-4107-8ad9-f3df17912b2c-bundle\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.552732 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552657    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxh2c\" (UniqueName: \"kubernetes.io/projected/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-kube-api-access-kxh2c\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.552732 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552672    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-bundle\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.552732 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552682    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc-util\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.552732 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552693    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46t4t\" (UniqueName: \"kubernetes.io/projected/50ced5a9-8dee-4107-8ad9-f3df17912b2c-kube-api-access-46t4t\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.553054 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.552872    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-bundle" (OuterVolumeSpecName: "bundle") pod "baf45629-8364-49a1-ace4-6d474ea305ce" (UID: "baf45629-8364-49a1-ace4-6d474ea305ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.553447 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.553418    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-bundle" (OuterVolumeSpecName: "bundle") pod "91b171c6-2841-49c2-a81b-33df9b575b99" (UID: "91b171c6-2841-49c2-a81b-33df9b575b99"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.554473 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.554445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b171c6-2841-49c2-a81b-33df9b575b99-kube-api-access-f4h9x" (OuterVolumeSpecName: "kube-api-access-f4h9x") pod "91b171c6-2841-49c2-a81b-33df9b575b99" (UID: "91b171c6-2841-49c2-a81b-33df9b575b99"). InnerVolumeSpecName "kube-api-access-f4h9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:59:30.554634 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.554611    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf45629-8364-49a1-ace4-6d474ea305ce-kube-api-access-pkvpc" (OuterVolumeSpecName: "kube-api-access-pkvpc") pod "baf45629-8364-49a1-ace4-6d474ea305ce" (UID: "baf45629-8364-49a1-ace4-6d474ea305ce"). InnerVolumeSpecName "kube-api-access-pkvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 21:59:30.560689 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.560665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-util" (OuterVolumeSpecName: "util") pod "baf45629-8364-49a1-ace4-6d474ea305ce" (UID: "baf45629-8364-49a1-ace4-6d474ea305ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.561422 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.561402    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-util" (OuterVolumeSpecName: "util") pod "91b171c6-2841-49c2-a81b-33df9b575b99" (UID: "91b171c6-2841-49c2-a81b-33df9b575b99"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 21:59:30.653782 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.653747    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-bundle\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.653782 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.653774    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-bundle\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.653782 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.653783    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkvpc\" (UniqueName: \"kubernetes.io/projected/baf45629-8364-49a1-ace4-6d474ea305ce-kube-api-access-pkvpc\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.654014 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.653793    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4h9x\" (UniqueName: \"kubernetes.io/projected/91b171c6-2841-49c2-a81b-33df9b575b99-kube-api-access-f4h9x\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.654014 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.653804    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf45629-8364-49a1-ace4-6d474ea305ce-util\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:30.654014 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:30.653811    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/91b171c6-2841-49c2-a81b-33df9b575b99-util\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 21:59:31.260700 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.260622    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j"
Sep 18 21:59:31.260861 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.260617    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2p72j" event={"ID":"50ced5a9-8dee-4107-8ad9-f3df17912b2c","Type":"ContainerDied","Data":"cca37d8515ebf0199d511b6892e44cd2e9d78a4181e405e8081ed3facac3bb26"}
Sep 18 21:59:31.260861 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.260749    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cca37d8515ebf0199d511b6892e44cd2e9d78a4181e405e8081ed3facac3bb26"
Sep 18 21:59:31.262663 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.262632    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg"
Sep 18 21:59:31.262804 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.262632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9pmg" event={"ID":"91b171c6-2841-49c2-a81b-33df9b575b99","Type":"ContainerDied","Data":"5bd6e0e5c62c450b5c36161c5c6a38571b33a4f89b09cf594b36d3f08d8ced38"}
Sep 18 21:59:31.262864 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.262799    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bd6e0e5c62c450b5c36161c5c6a38571b33a4f89b09cf594b36d3f08d8ced38"
Sep 18 21:59:31.264884 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.264861    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8"
Sep 18 21:59:31.265028 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.264861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9xs8" event={"ID":"baf45629-8364-49a1-ace4-6d474ea305ce","Type":"ContainerDied","Data":"53f1211574b06bc466b24feeb612000897023c03a671ab0b6c876b970003256f"}
Sep 18 21:59:31.265028 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.264916    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53f1211574b06bc466b24feeb612000897023c03a671ab0b6c876b970003256f"
Sep 18 21:59:31.267054 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.267031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn" event={"ID":"147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc","Type":"ContainerDied","Data":"18b1ba5c8e2eaf99ac148a98a1ec651e825ba6d6d950812fd7418ca186054e3f"}
Sep 18 21:59:31.267159 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.267064    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b1ba5c8e2eaf99ac148a98a1ec651e825ba6d6d950812fd7418ca186054e3f"
Sep 18 21:59:31.267159 ip-10-0-142-209 kubenswrapper[2582]: I0918 21:59:31.267079    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bnhqn"
Sep 18 22:00:01.567739 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.567707    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"]
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568060    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b171c6-2841-49c2-a81b-33df9b575b99" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568074    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b171c6-2841-49c2-a81b-33df9b575b99" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568087    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b171c6-2841-49c2-a81b-33df9b575b99" containerName="extract"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568092    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b171c6-2841-49c2-a81b-33df9b575b99" containerName="extract"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568099    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b171c6-2841-49c2-a81b-33df9b575b99" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568104    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b171c6-2841-49c2-a81b-33df9b575b99" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568113    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf45629-8364-49a1-ace4-6d474ea305ce" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568119    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf45629-8364-49a1-ace4-6d474ea305ce" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568129    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568133    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568140    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568145    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568150    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf45629-8364-49a1-ace4-6d474ea305ce" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568155    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf45629-8364-49a1-ace4-6d474ea305ce" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568161    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568166    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerName="pull"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568172    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568178    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerName="util"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568184    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerName="extract"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568189    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerName="extract"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568196    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerName="extract"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568200    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerName="extract"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568205    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf45629-8364-49a1-ace4-6d474ea305ce" containerName="extract"
Sep 18 22:00:01.568214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568210    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf45629-8364-49a1-ace4-6d474ea305ce" containerName="extract"
Sep 18 22:00:01.569009 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568262    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf45629-8364-49a1-ace4-6d474ea305ce" containerName="extract"
Sep 18 22:00:01.569009 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568268    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ced5a9-8dee-4107-8ad9-f3df17912b2c" containerName="extract"
Sep 18 22:00:01.569009 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568275    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b171c6-2841-49c2-a81b-33df9b575b99" containerName="extract"
Sep 18 22:00:01.569009 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.568281    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="147a4f4b-5e31-4f0c-a4e7-2b4cdc78eecc" containerName="extract"
Sep 18 22:00:01.572669 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.572653    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.575939 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.575916    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 22:00:01.576034 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.575999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 22:00:01.576099 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.576080    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 22:00:01.576809 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.576793    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wmx9d\""
Sep 18 22:00:01.582577 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.582562    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 22:00:01.588303 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.588278    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"]
Sep 18 22:00:01.695023 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.694989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.695203 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.695070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fqg\" (UniqueName: \"kubernetes.io/projected/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-kube-api-access-n4fqg\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.695203 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.695122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.796283 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.796250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fqg\" (UniqueName: \"kubernetes.io/projected/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-kube-api-access-n4fqg\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.796466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.796299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.796466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.796330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.796936 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.796915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.798760 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.798737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.805503 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.805483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fqg\" (UniqueName: \"kubernetes.io/projected/be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9-kube-api-access-n4fqg\") pod \"kuadrant-console-plugin-fbdd4f445-csmlv\" (UID: \"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:01.881793 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:01.881748    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"
Sep 18 22:00:02.172087 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:02.172064    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv"]
Sep 18 22:00:02.174778 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:00:02.174748    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe78e7a8_a0a1_46ab_8765_d0b0c9aefaf9.slice/crio-1c556626f48e8810fff79d0ddf997107477cbe406598bb184c72bd3ea966c589 WatchSource:0}: Error finding container 1c556626f48e8810fff79d0ddf997107477cbe406598bb184c72bd3ea966c589: Status 404 returned error can't find the container with id 1c556626f48e8810fff79d0ddf997107477cbe406598bb184c72bd3ea966c589
Sep 18 22:00:02.415017 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:02.414981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv" event={"ID":"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9","Type":"ContainerStarted","Data":"1c556626f48e8810fff79d0ddf997107477cbe406598bb184c72bd3ea966c589"}
Sep 18 22:00:23.535937 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:23.535893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv" event={"ID":"be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9","Type":"ContainerStarted","Data":"951be2f2f7b3b1e272309e11308ac72834e5542e334cc230bbc3744dc6330074"}
Sep 18 22:00:23.571114 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:23.571065    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-csmlv" podStartSLOduration=1.361110155 podStartE2EDuration="22.571049848s" totalImagesPullingTime="21.209939693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:23.570492613 +0000 UTC m=+580.603671507" watchObservedRunningTime="2026-09-18 22:00:23.571049848 +0000 UTC m=+580.604228752"
Sep 18 22:00:44.896492 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:44.896463    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:00:44.898388 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:44.898355    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:00:44.902852 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:44.902829    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:00:44.904761 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:44.904737    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:00:45.354566 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.354495    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lzrpp"]
Sep 18 22:00:45.363048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.363024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp"
Sep 18 22:00:45.366309 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.366287    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xkzfw\""
Sep 18 22:00:45.369900 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.369872    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lzrpp"]
Sep 18 22:00:45.398683 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.398656    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-48jgc"]
Sep 18 22:00:45.403767 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.403742    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-48jgc"]
Sep 18 22:00:45.403914 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.403826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-48jgc"
Sep 18 22:00:45.463632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.463606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2tv\" (UniqueName: \"kubernetes.io/projected/9586259b-a1bd-47cc-871f-dea48bb1dcf2-kube-api-access-6w2tv\") pod \"authorino-59c974bd48-48jgc\" (UID: \"9586259b-a1bd-47cc-871f-dea48bb1dcf2\") " pod="kuadrant-system/authorino-59c974bd48-48jgc"
Sep 18 22:00:45.463777 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.463659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxk4t\" (UniqueName: \"kubernetes.io/projected/a038118b-9487-40a7-abe3-2e09af72077b-kube-api-access-mxk4t\") pod \"authorino-7bcfd8f9cd-lzrpp\" (UID: \"a038118b-9487-40a7-abe3-2e09af72077b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp"
Sep 18 22:00:45.564384 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.564352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2tv\" (UniqueName: \"kubernetes.io/projected/9586259b-a1bd-47cc-871f-dea48bb1dcf2-kube-api-access-6w2tv\") pod \"authorino-59c974bd48-48jgc\" (UID: \"9586259b-a1bd-47cc-871f-dea48bb1dcf2\") " pod="kuadrant-system/authorino-59c974bd48-48jgc"
Sep 18 22:00:45.564539 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.564406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxk4t\" (UniqueName: \"kubernetes.io/projected/a038118b-9487-40a7-abe3-2e09af72077b-kube-api-access-mxk4t\") pod \"authorino-7bcfd8f9cd-lzrpp\" (UID: \"a038118b-9487-40a7-abe3-2e09af72077b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp"
Sep 18 22:00:45.573080 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.573044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxk4t\" (UniqueName: \"kubernetes.io/projected/a038118b-9487-40a7-abe3-2e09af72077b-kube-api-access-mxk4t\") pod \"authorino-7bcfd8f9cd-lzrpp\" (UID: \"a038118b-9487-40a7-abe3-2e09af72077b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp"
Sep 18 22:00:45.573080 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.573061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2tv\" (UniqueName: \"kubernetes.io/projected/9586259b-a1bd-47cc-871f-dea48bb1dcf2-kube-api-access-6w2tv\") pod \"authorino-59c974bd48-48jgc\" (UID: \"9586259b-a1bd-47cc-871f-dea48bb1dcf2\") " pod="kuadrant-system/authorino-59c974bd48-48jgc"
Sep 18 22:00:45.673100 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.673070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp"
Sep 18 22:00:45.713614 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.713584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-48jgc"
Sep 18 22:00:45.965322 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:45.965298    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lzrpp"]
Sep 18 22:00:45.966884 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:00:45.966856    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda038118b_9487_40a7_abe3_2e09af72077b.slice/crio-24517ef2a03e3111ea71af43cd00f87c8360e84d4eaea9a883a94df139446a63 WatchSource:0}: Error finding container 24517ef2a03e3111ea71af43cd00f87c8360e84d4eaea9a883a94df139446a63: Status 404 returned error can't find the container with id 24517ef2a03e3111ea71af43cd00f87c8360e84d4eaea9a883a94df139446a63
Sep 18 22:00:46.009617 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:46.009594    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-48jgc"]
Sep 18 22:00:46.010667 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:00:46.010644    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9586259b_a1bd_47cc_871f_dea48bb1dcf2.slice/crio-2a301e3f81d5d57073068ebfdaa58d255cc836f984c4a873a44f0feeccc85ed4 WatchSource:0}: Error finding container 2a301e3f81d5d57073068ebfdaa58d255cc836f984c4a873a44f0feeccc85ed4: Status 404 returned error can't find the container with id 2a301e3f81d5d57073068ebfdaa58d255cc836f984c4a873a44f0feeccc85ed4
Sep 18 22:00:46.649554 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:46.649518    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-48jgc" event={"ID":"9586259b-a1bd-47cc-871f-dea48bb1dcf2","Type":"ContainerStarted","Data":"2a301e3f81d5d57073068ebfdaa58d255cc836f984c4a873a44f0feeccc85ed4"}
Sep 18 22:00:46.650702 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:46.650668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp" event={"ID":"a038118b-9487-40a7-abe3-2e09af72077b","Type":"ContainerStarted","Data":"24517ef2a03e3111ea71af43cd00f87c8360e84d4eaea9a883a94df139446a63"}
Sep 18 22:00:48.663150 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:48.663104    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-48jgc" event={"ID":"9586259b-a1bd-47cc-871f-dea48bb1dcf2","Type":"ContainerStarted","Data":"28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9"}
Sep 18 22:00:48.670254 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:48.670216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp" event={"ID":"a038118b-9487-40a7-abe3-2e09af72077b","Type":"ContainerStarted","Data":"5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5"}
Sep 18 22:00:48.714366 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:48.714323    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-48jgc" podStartSLOduration=1.8369342290000001 podStartE2EDuration="3.714309737s" totalImagesPullingTime="1.877375508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:48.713296067 +0000 UTC m=+605.746474960" watchObservedRunningTime="2026-09-18 22:00:48.714309737 +0000 UTC m=+605.747488625"
Sep 18 22:00:48.750528 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:48.750478    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp" podStartSLOduration=1.834294911 podStartE2EDuration="3.750461117s" totalImagesPullingTime="1.916166206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:00:48.748903572 +0000 UTC m=+605.782082465" watchObservedRunningTime="2026-09-18 22:00:48.750461117 +0000 UTC m=+605.783640011"
Sep 18 22:00:48.763187 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:48.763158    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lzrpp"]
Sep 18 22:00:50.680369 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:50.680325    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp" podUID="a038118b-9487-40a7-abe3-2e09af72077b" containerName="authorino" containerID="cri-o://5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5" gracePeriod=30
Sep 18 22:00:50.920614 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:50.920590    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp"
Sep 18 22:00:51.009311 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.009238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxk4t\" (UniqueName: \"kubernetes.io/projected/a038118b-9487-40a7-abe3-2e09af72077b-kube-api-access-mxk4t\") pod \"a038118b-9487-40a7-abe3-2e09af72077b\" (UID: \"a038118b-9487-40a7-abe3-2e09af72077b\") "
Sep 18 22:00:51.011548 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.011523    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a038118b-9487-40a7-abe3-2e09af72077b-kube-api-access-mxk4t" (OuterVolumeSpecName: "kube-api-access-mxk4t") pod "a038118b-9487-40a7-abe3-2e09af72077b" (UID: "a038118b-9487-40a7-abe3-2e09af72077b"). InnerVolumeSpecName "kube-api-access-mxk4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:00:51.111008 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.110980    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxk4t\" (UniqueName: \"kubernetes.io/projected/a038118b-9487-40a7-abe3-2e09af72077b-kube-api-access-mxk4t\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:00:51.687315 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.687279    2582 generic.go:356] "Generic (PLEG): container finished" podID="a038118b-9487-40a7-abe3-2e09af72077b" containerID="5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5" exitCode=0
Sep 18 22:00:51.687315 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.687320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp"
Sep 18 22:00:51.687774 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.687362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp" event={"ID":"a038118b-9487-40a7-abe3-2e09af72077b","Type":"ContainerDied","Data":"5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5"}
Sep 18 22:00:51.687774 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.687411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-lzrpp" event={"ID":"a038118b-9487-40a7-abe3-2e09af72077b","Type":"ContainerDied","Data":"24517ef2a03e3111ea71af43cd00f87c8360e84d4eaea9a883a94df139446a63"}
Sep 18 22:00:51.687774 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.687428    2582 scope.go:117] "RemoveContainer" containerID="5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5"
Sep 18 22:00:51.696441 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.696424    2582 scope.go:117] "RemoveContainer" containerID="5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5"
Sep 18 22:00:51.696716 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:00:51.696696    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5\": container with ID starting with 5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5 not found: ID does not exist" containerID="5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5"
Sep 18 22:00:51.696784 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.696730    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5"} err="failed to get container status \"5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5\": rpc error: code = NotFound desc = could not find container \"5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5\": container with ID starting with 5cbebe64b1896059ac92e2b6efe86f5f93921483eefbc22250ca29484bed6bb5 not found: ID does not exist"
Sep 18 22:00:51.712499 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.712474    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lzrpp"]
Sep 18 22:00:51.714335 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:51.714315    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-lzrpp"]
Sep 18 22:00:52.974675 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:00:52.974645    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a038118b-9487-40a7-abe3-2e09af72077b" path="/var/lib/kubelet/pods/a038118b-9487-40a7-abe3-2e09af72077b/volumes"
Sep 18 22:01:10.207069 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.206992    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-48jgc"]
Sep 18 22:01:10.207541 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.207156    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-48jgc" podUID="9586259b-a1bd-47cc-871f-dea48bb1dcf2" containerName="authorino" containerID="cri-o://28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9" gracePeriod=30
Sep 18 22:01:10.454916 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.454893    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-48jgc"
Sep 18 22:01:10.567402 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.567321    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w2tv\" (UniqueName: \"kubernetes.io/projected/9586259b-a1bd-47cc-871f-dea48bb1dcf2-kube-api-access-6w2tv\") pod \"9586259b-a1bd-47cc-871f-dea48bb1dcf2\" (UID: \"9586259b-a1bd-47cc-871f-dea48bb1dcf2\") "
Sep 18 22:01:10.569319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.569296    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9586259b-a1bd-47cc-871f-dea48bb1dcf2-kube-api-access-6w2tv" (OuterVolumeSpecName: "kube-api-access-6w2tv") pod "9586259b-a1bd-47cc-871f-dea48bb1dcf2" (UID: "9586259b-a1bd-47cc-871f-dea48bb1dcf2"). InnerVolumeSpecName "kube-api-access-6w2tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:01:10.667917 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.667880    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2tv\" (UniqueName: \"kubernetes.io/projected/9586259b-a1bd-47cc-871f-dea48bb1dcf2-kube-api-access-6w2tv\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:01:10.783246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.783206    2582 generic.go:356] "Generic (PLEG): container finished" podID="9586259b-a1bd-47cc-871f-dea48bb1dcf2" containerID="28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9" exitCode=0
Sep 18 22:01:10.783417 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.783250    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-48jgc"
Sep 18 22:01:10.783417 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.783283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-48jgc" event={"ID":"9586259b-a1bd-47cc-871f-dea48bb1dcf2","Type":"ContainerDied","Data":"28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9"}
Sep 18 22:01:10.783417 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.783323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-48jgc" event={"ID":"9586259b-a1bd-47cc-871f-dea48bb1dcf2","Type":"ContainerDied","Data":"2a301e3f81d5d57073068ebfdaa58d255cc836f984c4a873a44f0feeccc85ed4"}
Sep 18 22:01:10.783417 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.783341    2582 scope.go:117] "RemoveContainer" containerID="28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9"
Sep 18 22:01:10.792586 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.792571    2582 scope.go:117] "RemoveContainer" containerID="28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9"
Sep 18 22:01:10.792861 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:01:10.792842    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9\": container with ID starting with 28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9 not found: ID does not exist" containerID="28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9"
Sep 18 22:01:10.792906 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.792873    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9"} err="failed to get container status \"28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9\": rpc error: code = NotFound desc = could not find container \"28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9\": container with ID starting with 28aaa7b2c8de9a4220c60d028e6e9d015ff804881e54e058bc286c655dd50fd9 not found: ID does not exist"
Sep 18 22:01:10.811797 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.811772    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-48jgc"]
Sep 18 22:01:10.820077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.820022    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-48jgc"]
Sep 18 22:01:10.974138 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:10.974102    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9586259b-a1bd-47cc-871f-dea48bb1dcf2" path="/var/lib/kubelet/pods/9586259b-a1bd-47cc-871f-dea48bb1dcf2/volumes"
Sep 18 22:01:45.619918 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.619873    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-8fqmw"]
Sep 18 22:01:45.620459 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.620440    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9586259b-a1bd-47cc-871f-dea48bb1dcf2" containerName="authorino"
Sep 18 22:01:45.620509 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.620463    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9586259b-a1bd-47cc-871f-dea48bb1dcf2" containerName="authorino"
Sep 18 22:01:45.620509 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.620484    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a038118b-9487-40a7-abe3-2e09af72077b" containerName="authorino"
Sep 18 22:01:45.620509 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.620493    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a038118b-9487-40a7-abe3-2e09af72077b" containerName="authorino"
Sep 18 22:01:45.620609 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.620566    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a038118b-9487-40a7-abe3-2e09af72077b" containerName="authorino"
Sep 18 22:01:45.620609 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.620582    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9586259b-a1bd-47cc-871f-dea48bb1dcf2" containerName="authorino"
Sep 18 22:01:45.625200 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.625178    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:45.628112 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.628088    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q6x8x\""
Sep 18 22:01:45.628246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.628151    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 22:01:45.628246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.628156    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 22:01:45.628754 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.628740    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 22:01:45.634481 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.634461    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8fqmw"]
Sep 18 22:01:45.641466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.641443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkm2\" (UniqueName: \"kubernetes.io/projected/d6a16208-ebef-4fce-982c-43e7551359b4-kube-api-access-nzkm2\") pod \"seaweedfs-64568bcd49-8fqmw\" (UID: \"d6a16208-ebef-4fce-982c-43e7551359b4\") " pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:45.641560 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.641499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6a16208-ebef-4fce-982c-43e7551359b4-data\") pod \"seaweedfs-64568bcd49-8fqmw\" (UID: \"d6a16208-ebef-4fce-982c-43e7551359b4\") " pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:45.742378 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.742351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkm2\" (UniqueName: \"kubernetes.io/projected/d6a16208-ebef-4fce-982c-43e7551359b4-kube-api-access-nzkm2\") pod \"seaweedfs-64568bcd49-8fqmw\" (UID: \"d6a16208-ebef-4fce-982c-43e7551359b4\") " pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:45.742583 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.742394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6a16208-ebef-4fce-982c-43e7551359b4-data\") pod \"seaweedfs-64568bcd49-8fqmw\" (UID: \"d6a16208-ebef-4fce-982c-43e7551359b4\") " pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:45.742720 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.742704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d6a16208-ebef-4fce-982c-43e7551359b4-data\") pod \"seaweedfs-64568bcd49-8fqmw\" (UID: \"d6a16208-ebef-4fce-982c-43e7551359b4\") " pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:45.774309 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.774280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkm2\" (UniqueName: \"kubernetes.io/projected/d6a16208-ebef-4fce-982c-43e7551359b4-kube-api-access-nzkm2\") pod \"seaweedfs-64568bcd49-8fqmw\" (UID: \"d6a16208-ebef-4fce-982c-43e7551359b4\") " pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:45.934730 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:45.934653    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:46.217118 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:46.217045    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-8fqmw"]
Sep 18 22:01:46.220511 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:01:46.220477    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a16208_ebef_4fce_982c_43e7551359b4.slice/crio-13baf4946518cb04f732510fc43b285e8b90621984a342dab3ec2b3eef0fe7c9 WatchSource:0}: Error finding container 13baf4946518cb04f732510fc43b285e8b90621984a342dab3ec2b3eef0fe7c9: Status 404 returned error can't find the container with id 13baf4946518cb04f732510fc43b285e8b90621984a342dab3ec2b3eef0fe7c9
Sep 18 22:01:46.988163 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:46.988123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8fqmw" event={"ID":"d6a16208-ebef-4fce-982c-43e7551359b4","Type":"ContainerStarted","Data":"13baf4946518cb04f732510fc43b285e8b90621984a342dab3ec2b3eef0fe7c9"}
Sep 18 22:01:50.005378 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:50.005335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-8fqmw" event={"ID":"d6a16208-ebef-4fce-982c-43e7551359b4","Type":"ContainerStarted","Data":"6ed332e8ddbcd9fccf0e18283c17f06eab2612ae3d6f01c335864fbc84bb80fb"}
Sep 18 22:01:50.005745 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:50.005409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:01:50.025772 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:50.025725    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-8fqmw" podStartSLOduration=1.881035635 podStartE2EDuration="5.025709548s" totalImagesPullingTime="3.144673913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:01:50.024755526 +0000 UTC m=+667.057934423" watchObservedRunningTime="2026-09-18 22:01:50.025709548 +0000 UTC m=+667.058888441"
Sep 18 22:01:56.012094 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:01:56.012065    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-8fqmw"
Sep 18 22:03:05.920629 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.920597    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-kv7l2"]
Sep 18 22:03:05.923972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.923940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:05.927712 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.927690    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 22:03:05.928016 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.927990    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5lngb\""
Sep 18 22:03:05.928215 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.928197    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 22:03:05.957432 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.957399    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-kv7l2"]
Sep 18 22:03:05.998101 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.998069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-cert\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:05.998271 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.998122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbk29\" (UniqueName: \"kubernetes.io/projected/42449c59-91d2-4d2a-93aa-efb5a91da3f1-kube-api-access-hbk29\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:05.998271 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:05.998245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-metrics-certs\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.099091 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.099051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-metrics-certs\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.099294 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.099109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-cert\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.099294 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.099141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbk29\" (UniqueName: \"kubernetes.io/projected/42449c59-91d2-4d2a-93aa-efb5a91da3f1-kube-api-access-hbk29\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.099294 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:03:06.099150    2582 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 18 22:03:06.099294 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:03:06.099210    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-metrics-certs podName:42449c59-91d2-4d2a-93aa-efb5a91da3f1 nodeName:}" failed. No retries permitted until 2026-09-18 22:03:06.599192652 +0000 UTC m=+743.632371530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-metrics-certs") pod "odh-model-controller-7b966d7984-kv7l2" (UID: "42449c59-91d2-4d2a-93aa-efb5a91da3f1") : secret "odh-model-controller-metrics-tls" not found
Sep 18 22:03:06.101669 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.101645    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-cert\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.111979 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.111957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbk29\" (UniqueName: \"kubernetes.io/projected/42449c59-91d2-4d2a-93aa-efb5a91da3f1-kube-api-access-hbk29\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.601928 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.601893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-metrics-certs\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.604418 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.604386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42449c59-91d2-4d2a-93aa-efb5a91da3f1-metrics-certs\") pod \"odh-model-controller-7b966d7984-kv7l2\" (UID: \"42449c59-91d2-4d2a-93aa-efb5a91da3f1\") " pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:06.834710 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:06.834670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:07.112995 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:07.112967    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-kv7l2"]
Sep 18 22:03:07.115783 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:03:07.115755    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42449c59_91d2_4d2a_93aa_efb5a91da3f1.slice/crio-d0c7d093bed9db4d4e3b8951f4b65cf9dcacfc5c415de2cd60998247a51e7dbf WatchSource:0}: Error finding container d0c7d093bed9db4d4e3b8951f4b65cf9dcacfc5c415de2cd60998247a51e7dbf: Status 404 returned error can't find the container with id d0c7d093bed9db4d4e3b8951f4b65cf9dcacfc5c415de2cd60998247a51e7dbf
Sep 18 22:03:07.117194 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:07.117174    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:03:07.395647 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:07.395611    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-kv7l2" event={"ID":"42449c59-91d2-4d2a-93aa-efb5a91da3f1","Type":"ContainerStarted","Data":"d0c7d093bed9db4d4e3b8951f4b65cf9dcacfc5c415de2cd60998247a51e7dbf"}
Sep 18 22:03:10.412527 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:10.412489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-kv7l2" event={"ID":"42449c59-91d2-4d2a-93aa-efb5a91da3f1","Type":"ContainerStarted","Data":"810fae2521dfbf69074bed22d4be8282c205cb7826536a5bd35b8c89370b0e4a"}
Sep 18 22:03:10.412940 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:10.412590    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:10.442757 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:10.442708    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-kv7l2" podStartSLOduration=2.801094232 podStartE2EDuration="5.442691242s" totalImagesPullingTime="2.64159701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:03:10.440011999 +0000 UTC m=+747.473190892" watchObservedRunningTime="2026-09-18 22:03:10.442691242 +0000 UTC m=+747.475870135"
Sep 18 22:03:21.420391 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:21.420362    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-kv7l2"
Sep 18 22:03:22.286418 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.286383    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9jbt8"]
Sep 18 22:03:22.290080 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.290063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.292513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.292484    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 22:03:22.303741 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.303719    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9jbt8"]
Sep 18 22:03:22.437381 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.437343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vc4\" (UniqueName: \"kubernetes.io/projected/21daa3ca-74a0-4198-b928-75d738e85c9c-kube-api-access-k4vc4\") pod \"s3-init-9jbt8\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") " pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.437741 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.437398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21daa3ca-74a0-4198-b928-75d738e85c9c-model-cache\") pod \"s3-init-9jbt8\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") " pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.538749 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.538715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vc4\" (UniqueName: \"kubernetes.io/projected/21daa3ca-74a0-4198-b928-75d738e85c9c-kube-api-access-k4vc4\") pod \"s3-init-9jbt8\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") " pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.538924 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.538772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21daa3ca-74a0-4198-b928-75d738e85c9c-model-cache\") pod \"s3-init-9jbt8\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") " pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.539132 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.539116    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21daa3ca-74a0-4198-b928-75d738e85c9c-model-cache\") pod \"s3-init-9jbt8\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") " pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.548501 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.548472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vc4\" (UniqueName: \"kubernetes.io/projected/21daa3ca-74a0-4198-b928-75d738e85c9c-kube-api-access-k4vc4\") pod \"s3-init-9jbt8\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") " pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.598877 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.598852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9jbt8"
Sep 18 22:03:22.897896 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:22.897869    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9jbt8"]
Sep 18 22:03:22.900465 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:03:22.900436    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21daa3ca_74a0_4198_b928_75d738e85c9c.slice/crio-6292f2cc0c1a95415a98ef7f50bdd658d4ae85a25cdd8d9ae24fe473d82149ef WatchSource:0}: Error finding container 6292f2cc0c1a95415a98ef7f50bdd658d4ae85a25cdd8d9ae24fe473d82149ef: Status 404 returned error can't find the container with id 6292f2cc0c1a95415a98ef7f50bdd658d4ae85a25cdd8d9ae24fe473d82149ef
Sep 18 22:03:23.484636 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:23.484600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9jbt8" event={"ID":"21daa3ca-74a0-4198-b928-75d738e85c9c","Type":"ContainerStarted","Data":"6292f2cc0c1a95415a98ef7f50bdd658d4ae85a25cdd8d9ae24fe473d82149ef"}
Sep 18 22:03:27.511694 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:27.511654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9jbt8" event={"ID":"21daa3ca-74a0-4198-b928-75d738e85c9c","Type":"ContainerStarted","Data":"776bba6f29739728d69fbea564dbd5f53c8b7c627a6298dd11fd7f0632ee163a"}
Sep 18 22:03:29.524568 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:29.524533    2582 generic.go:356] "Generic (PLEG): container finished" podID="21daa3ca-74a0-4198-b928-75d738e85c9c" containerID="776bba6f29739728d69fbea564dbd5f53c8b7c627a6298dd11fd7f0632ee163a" exitCode=0
Sep 18 22:03:29.524940 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:29.524607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9jbt8" event={"ID":"21daa3ca-74a0-4198-b928-75d738e85c9c","Type":"ContainerDied","Data":"776bba6f29739728d69fbea564dbd5f53c8b7c627a6298dd11fd7f0632ee163a"}
Sep 18 22:03:34.558622 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:34.558590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9jbt8" event={"ID":"21daa3ca-74a0-4198-b928-75d738e85c9c","Type":"ContainerStarted","Data":"f59240a61276c62654c4617d981145744e694019f42874b2fe4fe0dcb03d0b3f"}
Sep 18 22:03:34.583997 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:03:34.583931    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9jbt8" podStartSLOduration=2.554766261 podStartE2EDuration="12.583909882s" totalImagesPullingTime="8.357686718s" totalInitContainerRuntime="1.671456903s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:03:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:03:34.581626864 +0000 UTC m=+771.614805770" watchObservedRunningTime="2026-09-18 22:03:34.583909882 +0000 UTC m=+771.617088777"
Sep 18 22:04:39.881808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:39.881771    2582 generic.go:356] "Generic (PLEG): container finished" podID="21daa3ca-74a0-4198-b928-75d738e85c9c" containerID="f59240a61276c62654c4617d981145744e694019f42874b2fe4fe0dcb03d0b3f" exitCode=0
Sep 18 22:04:39.882242 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:39.881842    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9jbt8" event={"ID":"21daa3ca-74a0-4198-b928-75d738e85c9c","Type":"ContainerDied","Data":"f59240a61276c62654c4617d981145744e694019f42874b2fe4fe0dcb03d0b3f"}
Sep 18 22:04:41.016184 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.016159    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9jbt8"
Sep 18 22:04:41.195492 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.195409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4vc4\" (UniqueName: \"kubernetes.io/projected/21daa3ca-74a0-4198-b928-75d738e85c9c-kube-api-access-k4vc4\") pod \"21daa3ca-74a0-4198-b928-75d738e85c9c\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") "
Sep 18 22:04:41.195492 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.195445    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21daa3ca-74a0-4198-b928-75d738e85c9c-model-cache\") pod \"21daa3ca-74a0-4198-b928-75d738e85c9c\" (UID: \"21daa3ca-74a0-4198-b928-75d738e85c9c\") "
Sep 18 22:04:41.197452 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.197430    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21daa3ca-74a0-4198-b928-75d738e85c9c-kube-api-access-k4vc4" (OuterVolumeSpecName: "kube-api-access-k4vc4") pod "21daa3ca-74a0-4198-b928-75d738e85c9c" (UID: "21daa3ca-74a0-4198-b928-75d738e85c9c"). InnerVolumeSpecName "kube-api-access-k4vc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:04:41.218389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.218356    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21daa3ca-74a0-4198-b928-75d738e85c9c-model-cache" (OuterVolumeSpecName: "model-cache") pod "21daa3ca-74a0-4198-b928-75d738e85c9c" (UID: "21daa3ca-74a0-4198-b928-75d738e85c9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:04:41.296911 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.296874    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4vc4\" (UniqueName: \"kubernetes.io/projected/21daa3ca-74a0-4198-b928-75d738e85c9c-kube-api-access-k4vc4\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:04:41.296911 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.296905    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21daa3ca-74a0-4198-b928-75d738e85c9c-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:04:41.893205 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.893156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9jbt8" event={"ID":"21daa3ca-74a0-4198-b928-75d738e85c9c","Type":"ContainerDied","Data":"6292f2cc0c1a95415a98ef7f50bdd658d4ae85a25cdd8d9ae24fe473d82149ef"}
Sep 18 22:04:41.893205 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.893209    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6292f2cc0c1a95415a98ef7f50bdd658d4ae85a25cdd8d9ae24fe473d82149ef"
Sep 18 22:04:41.893401 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:04:41.893176    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9jbt8"
Sep 18 22:05:09.618020 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.617980    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"]
Sep 18 22:05:09.618545 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.618399    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21daa3ca-74a0-4198-b928-75d738e85c9c" containerName="download-hf-model"
Sep 18 22:05:09.618545 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.618414    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21daa3ca-74a0-4198-b928-75d738e85c9c" containerName="download-hf-model"
Sep 18 22:05:09.618545 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.618445    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21daa3ca-74a0-4198-b928-75d738e85c9c" containerName="s3-init"
Sep 18 22:05:09.618545 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.618451    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21daa3ca-74a0-4198-b928-75d738e85c9c" containerName="s3-init"
Sep 18 22:05:09.618545 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.618511    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="21daa3ca-74a0-4198-b928-75d738e85c9c" containerName="s3-init"
Sep 18 22:05:09.621399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.621382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.623646 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.623622    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 22:05:09.624392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.624369    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 22:05:09.624500 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.624412    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-whrzw\""
Sep 18 22:05:09.624500 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.624431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 22:05:09.631863 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.631841    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"]
Sep 18 22:05:09.734351 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.734316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.734510 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.734364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.734510 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.734451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.734510 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.734490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4gm\" (UniqueName: \"kubernetes.io/projected/67ff4988-c560-4c46-899e-c02119d6e51c-kube-api-access-bs4gm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.734626 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.734512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.734626 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.734578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff4988-c560-4c46-899e-c02119d6e51c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.834974 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.834928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4gm\" (UniqueName: \"kubernetes.io/projected/67ff4988-c560-4c46-899e-c02119d6e51c-kube-api-access-bs4gm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.834974 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.834978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.835225 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.835008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff4988-c560-4c46-899e-c02119d6e51c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.835225 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.835052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.835225 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.835072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.835225 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.835108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.835483 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.835461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.835542 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.835519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.835542 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.835529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.837362 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.837333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.837519 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.837503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff4988-c560-4c46-899e-c02119d6e51c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.842772 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.842744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4gm\" (UniqueName: \"kubernetes.io/projected/67ff4988-c560-4c46-899e-c02119d6e51c-kube-api-access-bs4gm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-qd48s\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:09.933977 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:09.933881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:10.225736 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:05:10.225704    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ff4988_c560_4c46_899e_c02119d6e51c.slice/crio-98ab5f29e523c3251c695695621a0accbb1b2d071abdc3b9218d8b2e5ff0122d WatchSource:0}: Error finding container 98ab5f29e523c3251c695695621a0accbb1b2d071abdc3b9218d8b2e5ff0122d: Status 404 returned error can't find the container with id 98ab5f29e523c3251c695695621a0accbb1b2d071abdc3b9218d8b2e5ff0122d
Sep 18 22:05:10.226119 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:10.226092    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"]
Sep 18 22:05:11.044419 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:11.044379    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" event={"ID":"67ff4988-c560-4c46-899e-c02119d6e51c","Type":"ContainerStarted","Data":"98ab5f29e523c3251c695695621a0accbb1b2d071abdc3b9218d8b2e5ff0122d"}
Sep 18 22:05:12.051044 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:12.051005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" event={"ID":"67ff4988-c560-4c46-899e-c02119d6e51c","Type":"ContainerStarted","Data":"0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b"}
Sep 18 22:05:12.073143 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:12.073093    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" podStartSLOduration=1.425294337 podStartE2EDuration="3.073078595s" totalImagesPullingTime="1.647784258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:05:12.072682605 +0000 UTC m=+869.105861500" watchObservedRunningTime="2026-09-18 22:05:12.073078595 +0000 UTC m=+869.106257489"
Sep 18 22:05:12.934793 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:12.934750    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:19.934385 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:19.934346    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:19.947250 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:19.947221    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:20.106704 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:20.106668    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:21.403380 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.403343    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"]
Sep 18 22:05:21.407295 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.407272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.409847 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.409823    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 18 22:05:21.409973 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.409857    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 18 22:05:21.410458 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.410431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 18 22:05:21.410559 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.410461    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-kv92f\""
Sep 18 22:05:21.421094 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.421065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"]
Sep 18 22:05:21.535303 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.535273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-home\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.535464 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.535308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2v7l\" (UniqueName: \"kubernetes.io/projected/c6a78409-d427-4947-9b0f-03feb8d2a7b0-kube-api-access-l2v7l\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.535464 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.535326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.535464 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.535398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-dshm\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.535464 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.535446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.535606 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.535466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.636775 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.636745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-dshm\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.636935 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.636790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.636935 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.636914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.637060 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.637044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-home\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.637106 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.637080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2v7l\" (UniqueName: \"kubernetes.io/projected/c6a78409-d427-4947-9b0f-03feb8d2a7b0-kube-api-access-l2v7l\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.637147 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.637107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.637147 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.637114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.637395 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.637375    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-home\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.637502 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.637438    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.639108 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.639088    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-dshm\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.639274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.639260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.648297 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.648274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2v7l\" (UniqueName: \"kubernetes.io/projected/c6a78409-d427-4947-9b0f-03feb8d2a7b0-kube-api-access-l2v7l\") pod \"fc-smoke-test-kserve-7f4687897-xrvkz\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:21.732444 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:21.732335    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:22.013439 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:22.013414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"]
Sep 18 22:05:22.015074 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:05:22.015048    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a78409_d427_4947_9b0f_03feb8d2a7b0.slice/crio-518bca157cae87d7772822d7930715372c53681713c02d8a238aff5617dd6a53 WatchSource:0}: Error finding container 518bca157cae87d7772822d7930715372c53681713c02d8a238aff5617dd6a53: Status 404 returned error can't find the container with id 518bca157cae87d7772822d7930715372c53681713c02d8a238aff5617dd6a53
Sep 18 22:05:22.107163 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:22.107123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" event={"ID":"c6a78409-d427-4947-9b0f-03feb8d2a7b0","Type":"ContainerStarted","Data":"667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7"}
Sep 18 22:05:22.107344 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:22.107179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" event={"ID":"c6a78409-d427-4947-9b0f-03feb8d2a7b0","Type":"ContainerStarted","Data":"518bca157cae87d7772822d7930715372c53681713c02d8a238aff5617dd6a53"}
Sep 18 22:05:22.128398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:22.128348    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" podStartSLOduration=1.128331757 podStartE2EDuration="1.128331757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:05:22.126857943 +0000 UTC m=+879.160036847" watchObservedRunningTime="2026-09-18 22:05:22.128331757 +0000 UTC m=+879.161510644"
Sep 18 22:05:22.732456 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:22.732419    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:31.732842 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:31.732762    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:31.745361 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:31.745335    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:32.171021 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:32.170997    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:05:44.934116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:44.934085    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:05:44.938165 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:44.938141    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:05:44.940091 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:44.940073    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:05:44.944395 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:44.944376    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:05:53.048118 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.048082    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"]
Sep 18 22:05:53.057759 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:05:53.057724    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:05:53.057759 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:05:53.057759 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:05:53.057759 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b" cmd=["/bin/sleep","15"]
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.057780    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" message="" err=<
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:05:53.057791    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" podUID="67ff4988-c560-4c46-899e-c02119d6e51c" containerName="main" containerID="cri-o://0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b"
Sep 18 22:05:53.058019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.057836    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" podUID="67ff4988-c560-4c46-899e-c02119d6e51c" containerName="main" containerID="cri-o://0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b" gracePeriod=60
Sep 18 22:05:53.095962 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.095902    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" podUID="67ff4988-c560-4c46-899e-c02119d6e51c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 18 22:05:53.398027 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.397997    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:53.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.419997    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-tmp-dir\") pod \"67ff4988-c560-4c46-899e-c02119d6e51c\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") "
Sep 18 22:05:53.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.420049    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff4988-c560-4c46-899e-c02119d6e51c-tls-certs\") pod \"67ff4988-c560-4c46-899e-c02119d6e51c\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") "
Sep 18 22:05:53.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.420087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-model-cache\") pod \"67ff4988-c560-4c46-899e-c02119d6e51c\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") "
Sep 18 22:05:53.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.420129    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-home\") pod \"67ff4988-c560-4c46-899e-c02119d6e51c\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") "
Sep 18 22:05:53.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.420170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs4gm\" (UniqueName: \"kubernetes.io/projected/67ff4988-c560-4c46-899e-c02119d6e51c-kube-api-access-bs4gm\") pod \"67ff4988-c560-4c46-899e-c02119d6e51c\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") "
Sep 18 22:05:53.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.420218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-dshm\") pod \"67ff4988-c560-4c46-899e-c02119d6e51c\" (UID: \"67ff4988-c560-4c46-899e-c02119d6e51c\") "
Sep 18 22:05:53.423831 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.423720    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-model-cache" (OuterVolumeSpecName: "model-cache") pod "67ff4988-c560-4c46-899e-c02119d6e51c" (UID: "67ff4988-c560-4c46-899e-c02119d6e51c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:05:53.423940 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.423902    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67ff4988-c560-4c46-899e-c02119d6e51c" (UID: "67ff4988-c560-4c46-899e-c02119d6e51c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:05:53.425277 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.425182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-home" (OuterVolumeSpecName: "home") pod "67ff4988-c560-4c46-899e-c02119d6e51c" (UID: "67ff4988-c560-4c46-899e-c02119d6e51c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:05:53.430144 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.425305    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-dshm" (OuterVolumeSpecName: "dshm") pod "67ff4988-c560-4c46-899e-c02119d6e51c" (UID: "67ff4988-c560-4c46-899e-c02119d6e51c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:05:53.430144 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.426493    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ff4988-c560-4c46-899e-c02119d6e51c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67ff4988-c560-4c46-899e-c02119d6e51c" (UID: "67ff4988-c560-4c46-899e-c02119d6e51c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:05:53.430144 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.427563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ff4988-c560-4c46-899e-c02119d6e51c-kube-api-access-bs4gm" (OuterVolumeSpecName: "kube-api-access-bs4gm") pod "67ff4988-c560-4c46-899e-c02119d6e51c" (UID: "67ff4988-c560-4c46-899e-c02119d6e51c"). InnerVolumeSpecName "kube-api-access-bs4gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:05:53.521226 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.521198    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:05:53.521226 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.521220    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff4988-c560-4c46-899e-c02119d6e51c-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:05:53.521226 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.521228    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:05:53.521430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.521236    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:05:53.521430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.521244    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4gm\" (UniqueName: \"kubernetes.io/projected/67ff4988-c560-4c46-899e-c02119d6e51c-kube-api-access-bs4gm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:05:53.521430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:53.521252    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ff4988-c560-4c46-899e-c02119d6e51c-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:05:54.274098 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.274063    2582 generic.go:356] "Generic (PLEG): container finished" podID="67ff4988-c560-4c46-899e-c02119d6e51c" containerID="0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b" exitCode=0
Sep 18 22:05:54.274528 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.274123    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"
Sep 18 22:05:54.274528 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.274150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" event={"ID":"67ff4988-c560-4c46-899e-c02119d6e51c","Type":"ContainerDied","Data":"0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b"}
Sep 18 22:05:54.274528 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.274204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s" event={"ID":"67ff4988-c560-4c46-899e-c02119d6e51c","Type":"ContainerDied","Data":"98ab5f29e523c3251c695695621a0accbb1b2d071abdc3b9218d8b2e5ff0122d"}
Sep 18 22:05:54.274528 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.274226    2582 scope.go:117] "RemoveContainer" containerID="0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b"
Sep 18 22:05:54.283774 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.283759    2582 scope.go:117] "RemoveContainer" containerID="0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b"
Sep 18 22:05:54.284068 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:05:54.284049    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b\": container with ID starting with 0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b not found: ID does not exist" containerID="0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b"
Sep 18 22:05:54.284142 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.284075    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b"} err="failed to get container status \"0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b\": rpc error: code = NotFound desc = could not find container \"0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b\": container with ID starting with 0d4368cc128e10cd202baf38e05c1e6da8c25fad1071d7a2746868ded5795c2b not found: ID does not exist"
Sep 18 22:05:54.297919 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.297897    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"]
Sep 18 22:05:54.301101 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.301077    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-qd48s"]
Sep 18 22:05:54.973661 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:05:54.973627    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ff4988-c560-4c46-899e-c02119d6e51c" path="/var/lib/kubelet/pods/67ff4988-c560-4c46-899e-c02119d6e51c/volumes"
Sep 18 22:06:01.371435 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.371392    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"]
Sep 18 22:06:01.380473 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:06:01.380437    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:06:01.380473 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:06:01.380473 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:06:01.380473 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7" cmd=["/bin/sleep","15"]
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.380504    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" message="" err=<
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:06:01.380521    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" podUID="c6a78409-d427-4947-9b0f-03feb8d2a7b0" containerName="main" containerID="cri-o://667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7"
Sep 18 22:06:01.380782 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.380563    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" podUID="c6a78409-d427-4947-9b0f-03feb8d2a7b0" containerName="main" containerID="cri-o://667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7" gracePeriod=60
Sep 18 22:06:01.728400 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.728374    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:06:01.789069 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789039    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tmp-dir\") pod \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") "
Sep 18 22:06:01.789250 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789082    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-model-cache\") pod \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") "
Sep 18 22:06:01.789250 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789102    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-dshm\") pod \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") "
Sep 18 22:06:01.789250 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tls-certs\") pod \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") "
Sep 18 22:06:01.789250 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789176    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2v7l\" (UniqueName: \"kubernetes.io/projected/c6a78409-d427-4947-9b0f-03feb8d2a7b0-kube-api-access-l2v7l\") pod \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") "
Sep 18 22:06:01.789250 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789227    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-home\") pod \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\" (UID: \"c6a78409-d427-4947-9b0f-03feb8d2a7b0\") "
Sep 18 22:06:01.789513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789364    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6a78409-d427-4947-9b0f-03feb8d2a7b0" (UID: "c6a78409-d427-4947-9b0f-03feb8d2a7b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:06:01.789513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789371    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6a78409-d427-4947-9b0f-03feb8d2a7b0" (UID: "c6a78409-d427-4947-9b0f-03feb8d2a7b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:06:01.789513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789456    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:06:01.789513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789477    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:06:01.789696 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.789602    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-home" (OuterVolumeSpecName: "home") pod "c6a78409-d427-4947-9b0f-03feb8d2a7b0" (UID: "c6a78409-d427-4947-9b0f-03feb8d2a7b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:06:01.791594 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.791567    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-dshm" (OuterVolumeSpecName: "dshm") pod "c6a78409-d427-4947-9b0f-03feb8d2a7b0" (UID: "c6a78409-d427-4947-9b0f-03feb8d2a7b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:06:01.791708 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.791588    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a78409-d427-4947-9b0f-03feb8d2a7b0-kube-api-access-l2v7l" (OuterVolumeSpecName: "kube-api-access-l2v7l") pod "c6a78409-d427-4947-9b0f-03feb8d2a7b0" (UID: "c6a78409-d427-4947-9b0f-03feb8d2a7b0"). InnerVolumeSpecName "kube-api-access-l2v7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:06:01.791708 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.791573    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6a78409-d427-4947-9b0f-03feb8d2a7b0" (UID: "c6a78409-d427-4947-9b0f-03feb8d2a7b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:06:01.890354 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.890319    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:06:01.890354 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.890349    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a78409-d427-4947-9b0f-03feb8d2a7b0-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:06:01.890354 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.890362    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2v7l\" (UniqueName: \"kubernetes.io/projected/c6a78409-d427-4947-9b0f-03feb8d2a7b0-kube-api-access-l2v7l\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:06:01.890569 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:01.890371    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a78409-d427-4947-9b0f-03feb8d2a7b0-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:06:02.317023 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.316985    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6a78409-d427-4947-9b0f-03feb8d2a7b0" containerID="667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7" exitCode=0
Sep 18 22:06:02.317222 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.317051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" event={"ID":"c6a78409-d427-4947-9b0f-03feb8d2a7b0","Type":"ContainerDied","Data":"667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7"}
Sep 18 22:06:02.317222 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.317083    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"
Sep 18 22:06:02.317222 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.317098    2582 scope.go:117] "RemoveContainer" containerID="667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7"
Sep 18 22:06:02.317222 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.317089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz" event={"ID":"c6a78409-d427-4947-9b0f-03feb8d2a7b0","Type":"ContainerDied","Data":"518bca157cae87d7772822d7930715372c53681713c02d8a238aff5617dd6a53"}
Sep 18 22:06:02.326714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.326697    2582 scope.go:117] "RemoveContainer" containerID="667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7"
Sep 18 22:06:02.327035 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:06:02.327016    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7\": container with ID starting with 667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7 not found: ID does not exist" containerID="667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7"
Sep 18 22:06:02.327089 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.327044    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7"} err="failed to get container status \"667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7\": rpc error: code = NotFound desc = could not find container \"667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7\": container with ID starting with 667829b1d4864c3051e3ba4328d914c58a09cdc20f34eb8ca3e28f8e91b7a3d7 not found: ID does not exist"
Sep 18 22:06:02.338445 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.338420    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"]
Sep 18 22:06:02.344380 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.344357    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-xrvkz"]
Sep 18 22:06:02.973852 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:02.973816    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a78409-d427-4947-9b0f-03feb8d2a7b0" path="/var/lib/kubelet/pods/c6a78409-d427-4947-9b0f-03feb8d2a7b0/volumes"
Sep 18 22:06:49.606100 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.606067    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"]
Sep 18 22:06:49.606513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.606437    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ff4988-c560-4c46-899e-c02119d6e51c" containerName="main"
Sep 18 22:06:49.606513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.606449    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ff4988-c560-4c46-899e-c02119d6e51c" containerName="main"
Sep 18 22:06:49.606513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.606458    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a78409-d427-4947-9b0f-03feb8d2a7b0" containerName="main"
Sep 18 22:06:49.606513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.606463    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a78409-d427-4947-9b0f-03feb8d2a7b0" containerName="main"
Sep 18 22:06:49.606740 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.606531    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a78409-d427-4947-9b0f-03feb8d2a7b0" containerName="main"
Sep 18 22:06:49.606740 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.606542    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ff4988-c560-4c46-899e-c02119d6e51c" containerName="main"
Sep 18 22:06:49.610925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.610906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.614106 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.614080    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 22:06:49.614247 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.614086    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-cnsnk\""
Sep 18 22:06:49.614247 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.614169    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 22:06:49.614371 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.614326    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 22:06:49.620804 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.620781    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"]
Sep 18 22:06:49.686752 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.686716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.686904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.686761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltg9\" (UniqueName: \"kubernetes.io/projected/794c6392-4411-43f1-b752-361f0534b084-kube-api-access-cltg9\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.686904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.686812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.686904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.686859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.687100 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.686906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.687100 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.686932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794c6392-4411-43f1-b752-361f0534b084-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.787674 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.787641    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.787796 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.787689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltg9\" (UniqueName: \"kubernetes.io/projected/794c6392-4411-43f1-b752-361f0534b084-kube-api-access-cltg9\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.787796 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.787715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.787796 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.787748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.788004 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.787897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.788004 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.787970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794c6392-4411-43f1-b752-361f0534b084-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.788198 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.788171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.788296 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.788232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.788296 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.788271    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.789966 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.789935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.790246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.790229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794c6392-4411-43f1-b752-361f0534b084-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.795061 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.795043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltg9\" (UniqueName: \"kubernetes.io/projected/794c6392-4411-43f1-b752-361f0534b084-kube-api-access-cltg9\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-lvhjn\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:49.923234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:49.923203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:50.197423 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:50.197398    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"]
Sep 18 22:06:50.199443 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:06:50.199407    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794c6392_4411_43f1_b752_361f0534b084.slice/crio-ce348ab3ab441557eea86431085c9e779fef2ed9878f79bdfd276ba198ae51de WatchSource:0}: Error finding container ce348ab3ab441557eea86431085c9e779fef2ed9878f79bdfd276ba198ae51de: Status 404 returned error can't find the container with id ce348ab3ab441557eea86431085c9e779fef2ed9878f79bdfd276ba198ae51de
Sep 18 22:06:50.565681 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:50.565595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" event={"ID":"794c6392-4411-43f1-b752-361f0534b084","Type":"ContainerStarted","Data":"8ce453632597248d7d6a0c5fab9d4e516442fae3ed28b2e28cb83cfcb5f5cba0"}
Sep 18 22:06:50.565681 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:50.565652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" event={"ID":"794c6392-4411-43f1-b752-361f0534b084","Type":"ContainerStarted","Data":"ce348ab3ab441557eea86431085c9e779fef2ed9878f79bdfd276ba198ae51de"}
Sep 18 22:06:50.591839 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:50.591784    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" podStartSLOduration=1.591766861 podStartE2EDuration="1.591766861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:06:50.59029273 +0000 UTC m=+967.623471623" watchObservedRunningTime="2026-09-18 22:06:50.591766861 +0000 UTC m=+967.624945755"
Sep 18 22:06:50.923695 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:50.923655    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:59.924399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:59.924308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:06:59.936841 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:06:59.936816    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:07:00.629258 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:00.629226    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:07:40.539887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.539848    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"]
Sep 18 22:07:40.550186 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:07:40.550152    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:07:40.550186 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:07:40.550186 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:07:40.550186 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="8ce453632597248d7d6a0c5fab9d4e516442fae3ed28b2e28cb83cfcb5f5cba0" cmd=["/bin/sleep","15"]
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.550210    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" message="" err=<
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:07:40.550226    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" podUID="794c6392-4411-43f1-b752-361f0534b084" containerName="main" containerID="cri-o://8ce453632597248d7d6a0c5fab9d4e516442fae3ed28b2e28cb83cfcb5f5cba0"
Sep 18 22:07:40.550478 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.550283    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" podUID="794c6392-4411-43f1-b752-361f0534b084" containerName="main" containerID="cri-o://8ce453632597248d7d6a0c5fab9d4e516442fae3ed28b2e28cb83cfcb5f5cba0" gracePeriod=60
Sep 18 22:07:40.618857 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.618817    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" podUID="794c6392-4411-43f1-b752-361f0534b084" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 18 22:07:40.829078 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.829041    2582 generic.go:356] "Generic (PLEG): container finished" podID="794c6392-4411-43f1-b752-361f0534b084" containerID="8ce453632597248d7d6a0c5fab9d4e516442fae3ed28b2e28cb83cfcb5f5cba0" exitCode=0
Sep 18 22:07:40.829227 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.829085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" event={"ID":"794c6392-4411-43f1-b752-361f0534b084","Type":"ContainerDied","Data":"8ce453632597248d7d6a0c5fab9d4e516442fae3ed28b2e28cb83cfcb5f5cba0"}
Sep 18 22:07:40.895298 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.895273    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:07:40.927443 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927419    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-model-cache\") pod \"794c6392-4411-43f1-b752-361f0534b084\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") "
Sep 18 22:07:40.927587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927454    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-dshm\") pod \"794c6392-4411-43f1-b752-361f0534b084\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") "
Sep 18 22:07:40.927587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-home\") pod \"794c6392-4411-43f1-b752-361f0534b084\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") "
Sep 18 22:07:40.927587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794c6392-4411-43f1-b752-361f0534b084-tls-certs\") pod \"794c6392-4411-43f1-b752-361f0534b084\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") "
Sep 18 22:07:40.927587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-tmp-dir\") pod \"794c6392-4411-43f1-b752-361f0534b084\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") "
Sep 18 22:07:40.927587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cltg9\" (UniqueName: \"kubernetes.io/projected/794c6392-4411-43f1-b752-361f0534b084-kube-api-access-cltg9\") pod \"794c6392-4411-43f1-b752-361f0534b084\" (UID: \"794c6392-4411-43f1-b752-361f0534b084\") "
Sep 18 22:07:40.927857 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927714    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-home" (OuterVolumeSpecName: "home") pod "794c6392-4411-43f1-b752-361f0534b084" (UID: "794c6392-4411-43f1-b752-361f0534b084"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:07:40.927857 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927720    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-model-cache" (OuterVolumeSpecName: "model-cache") pod "794c6392-4411-43f1-b752-361f0534b084" (UID: "794c6392-4411-43f1-b752-361f0534b084"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:07:40.927857 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.927818    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "794c6392-4411-43f1-b752-361f0534b084" (UID: "794c6392-4411-43f1-b752-361f0534b084"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:07:40.929803 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.929773    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794c6392-4411-43f1-b752-361f0534b084-kube-api-access-cltg9" (OuterVolumeSpecName: "kube-api-access-cltg9") pod "794c6392-4411-43f1-b752-361f0534b084" (UID: "794c6392-4411-43f1-b752-361f0534b084"). InnerVolumeSpecName "kube-api-access-cltg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:07:40.930038 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.930010    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794c6392-4411-43f1-b752-361f0534b084-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "794c6392-4411-43f1-b752-361f0534b084" (UID: "794c6392-4411-43f1-b752-361f0534b084"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:07:40.930038 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:40.930028    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-dshm" (OuterVolumeSpecName: "dshm") pod "794c6392-4411-43f1-b752-361f0534b084" (UID: "794c6392-4411-43f1-b752-361f0534b084"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:07:41.028101 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.028073    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:07:41.028101 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.028098    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltg9\" (UniqueName: \"kubernetes.io/projected/794c6392-4411-43f1-b752-361f0534b084-kube-api-access-cltg9\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:07:41.028265 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.028108    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:07:41.028265 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.028116    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:07:41.028265 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.028123    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/794c6392-4411-43f1-b752-361f0534b084-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:07:41.028265 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.028133    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/794c6392-4411-43f1-b752-361f0534b084-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:07:41.836231 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.836191    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn" event={"ID":"794c6392-4411-43f1-b752-361f0534b084","Type":"ContainerDied","Data":"ce348ab3ab441557eea86431085c9e779fef2ed9878f79bdfd276ba198ae51de"}
Sep 18 22:07:41.836231 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.836222    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"
Sep 18 22:07:41.836780 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.836266    2582 scope.go:117] "RemoveContainer" containerID="8ce453632597248d7d6a0c5fab9d4e516442fae3ed28b2e28cb83cfcb5f5cba0"
Sep 18 22:07:41.854336 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.854310    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"]
Sep 18 22:07:41.857304 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:41.857282    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-lvhjn"]
Sep 18 22:07:42.973382 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:07:42.973352    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="794c6392-4411-43f1-b752-361f0534b084" path="/var/lib/kubelet/pods/794c6392-4411-43f1-b752-361f0534b084/volumes"
Sep 18 22:08:33.730122 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.730048    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"]
Sep 18 22:08:33.730512 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.730424    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="794c6392-4411-43f1-b752-361f0534b084" containerName="main"
Sep 18 22:08:33.730512 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.730437    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="794c6392-4411-43f1-b752-361f0534b084" containerName="main"
Sep 18 22:08:33.730512 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.730504    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="794c6392-4411-43f1-b752-361f0534b084" containerName="main"
Sep 18 22:08:33.733498 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.733483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.736532 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.736503    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-bcctn\""
Sep 18 22:08:33.736735 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.736585    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 22:08:33.736735 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.736597    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 22:08:33.736735 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.736514    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 22:08:33.736913 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.736777    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-d6hkm\""
Sep 18 22:08:33.743675 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.743656    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"]
Sep 18 22:08:33.857361 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.857326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.857522 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.857365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.857522 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.857396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzb5x\" (UniqueName: \"kubernetes.io/projected/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-kube-api-access-jzb5x\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.857522 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.857480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.857630 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.857521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.857630 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.857569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.958853 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.958821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959056 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.958879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959056 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.958900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959056 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.958918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzb5x\" (UniqueName: \"kubernetes.io/projected/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-kube-api-access-jzb5x\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959056 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.958990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959056 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.959012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.959369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959454 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.959386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.959454 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.959427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.961265 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.961241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.961542 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.961521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:33.969962 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:33.969926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzb5x\" (UniqueName: \"kubernetes.io/projected/e6c236b5-5165-46bf-a4b0-a9cbd13ca414-kube-api-access-jzb5x\") pod \"scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9\" (UID: \"e6c236b5-5165-46bf-a4b0-a9cbd13ca414\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:34.043416 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:34.043336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:34.320164 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:34.320102    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"]
Sep 18 22:08:34.322490 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:08:34.322468    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c236b5_5165_46bf_a4b0_a9cbd13ca414.slice/crio-2d9cc1ff92d0afa939e794d7e45ae99b1d914e091501efc01a58fc37c61f76e2 WatchSource:0}: Error finding container 2d9cc1ff92d0afa939e794d7e45ae99b1d914e091501efc01a58fc37c61f76e2: Status 404 returned error can't find the container with id 2d9cc1ff92d0afa939e794d7e45ae99b1d914e091501efc01a58fc37c61f76e2
Sep 18 22:08:34.324406 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:34.324389    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:08:35.115596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:35.115549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9" event={"ID":"e6c236b5-5165-46bf-a4b0-a9cbd13ca414","Type":"ContainerStarted","Data":"2d9cc1ff92d0afa939e794d7e45ae99b1d914e091501efc01a58fc37c61f76e2"}
Sep 18 22:08:36.122532 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:36.122452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9" event={"ID":"e6c236b5-5165-46bf-a4b0-a9cbd13ca414","Type":"ContainerStarted","Data":"54bb1fc56ff7a784d948b57f24ce57a599a7447a47b421328711de4f36642f8d"}
Sep 18 22:08:36.122894 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:36.122568    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:37.129665 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:37.129628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9" event={"ID":"e6c236b5-5165-46bf-a4b0-a9cbd13ca414","Type":"ContainerStarted","Data":"c54de8b367bb6d6e86e63ff4b129f89623387aefb2971c316cf4852fb53820fa"}
Sep 18 22:08:37.154633 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:37.154571    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9" podStartSLOduration=2.607314935 podStartE2EDuration="4.154550376s" totalImagesPullingTime="1.547235441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:08:37.152687344 +0000 UTC m=+1074.185866263" watchObservedRunningTime="2026-09-18 22:08:37.154550376 +0000 UTC m=+1074.187729271"
Sep 18 22:08:38.043690 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:38.043658    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:44.043690 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:44.043652    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:44.056039 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:44.056012    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:44.178333 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:44.178301    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:08:45.185248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:08:45.185219    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9"
Sep 18 22:10:44.972521 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:10:44.972490    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:10:44.978973 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:10:44.978940    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:10:44.979653 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:10:44.979631    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:10:44.985356 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:10:44.985339    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:15:45.009679 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:15:45.009649    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:15:45.015565 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:15:45.015542    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:15:45.017075 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:15:45.017056    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:15:45.022511 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:15:45.022494    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:20:45.047823 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:20:45.047725    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:20:45.054360 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:20:45.054336    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:20:45.056826 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:20:45.056805    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:20:45.062779 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:20:45.062763    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:21:53.423365 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.423325    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"]
Sep 18 22:21:53.426892 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.426870    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.429196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.429152    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 18 22:21:53.430065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.429880    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 18 22:21:53.430065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.429924    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-9wv8x\""
Sep 18 22:21:53.430065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.429974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 18 22:21:53.448687 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.448664    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"]
Sep 18 22:21:53.488155 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488289 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488289 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488289 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488289 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488433 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4sr\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-kube-api-access-dv4sr\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488433 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488433 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.488530 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.488434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cb57c8-be39-4412-aa05-8ea9082556d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589042 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.588995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cb57c8-be39-4412-aa05-8ea9082556d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589543 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4sr\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-kube-api-access-dv4sr\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589543 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589648 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589545    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589701 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589701 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589781 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.589972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.589927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cb57c8-be39-4412-aa05-8ea9082556d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.591462 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.591445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.591685 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.591667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.599196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.599175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.599319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.599299    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4sr\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-kube-api-access-dv4sr\") pod \"router-gateway-1-openshift-default-6b46477c5b-6xcpd\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:53.737829 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:53.737756    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:54.015722 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:54.015596    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"]
Sep 18 22:21:54.018852 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:21:54.018824    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cb57c8_be39_4412_aa05_8ea9082556d8.slice/crio-0b995b149b4afea03b10370cbc7146a88308e60850ff12633befb48386a30f0d WatchSource:0}: Error finding container 0b995b149b4afea03b10370cbc7146a88308e60850ff12633befb48386a30f0d: Status 404 returned error can't find the container with id 0b995b149b4afea03b10370cbc7146a88308e60850ff12633befb48386a30f0d
Sep 18 22:21:54.021570 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:54.021554    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:21:54.236328 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:54.236295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd" event={"ID":"a2cb57c8-be39-4412-aa05-8ea9082556d8","Type":"ContainerStarted","Data":"0b995b149b4afea03b10370cbc7146a88308e60850ff12633befb48386a30f0d"}
Sep 18 22:21:56.580302 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:56.580261    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:21:56.580553 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:56.580335    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:21:56.580553 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:56.580367    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:21:57.257446 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:57.257356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd" event={"ID":"a2cb57c8-be39-4412-aa05-8ea9082556d8","Type":"ContainerStarted","Data":"f84e54b3962a1a0514d9be084f89c42cb9f817aa6ca7aea5073fe9d7db33c117"}
Sep 18 22:21:57.278656 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:57.278607    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd" podStartSLOduration=1.720217449 podStartE2EDuration="4.278592446s" totalImagesPullingTime="2.558374997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:21:57.277030498 +0000 UTC m=+1874.310209392" watchObservedRunningTime="2026-09-18 22:21:57.278592446 +0000 UTC m=+1874.311771338"
Sep 18 22:21:57.738676 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:57.738644    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:57.744405 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:57.744376    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:58.262657 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:58.262620    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:21:58.263564 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:21:58.263542    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:22:18.137083 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.137044    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"]
Sep 18 22:22:18.141154 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.141137    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.144393 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.144371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-dc7v9\""
Sep 18 22:22:18.144506 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.144424    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 18 22:22:18.153563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.153543    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"]
Sep 18 22:22:18.307008 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.306968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014e29d1-f292-4299-8a8b-ea492b365077-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.307237 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.307028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.307237 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.307067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.307237 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.307170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.307237 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.307211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.307397 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.307260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m29m\" (UniqueName: \"kubernetes.io/projected/014e29d1-f292-4299-8a8b-ea492b365077-kube-api-access-2m29m\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408394 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408394 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m29m\" (UniqueName: \"kubernetes.io/projected/014e29d1-f292-4299-8a8b-ea492b365077-kube-api-access-2m29m\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408394 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014e29d1-f292-4299-8a8b-ea492b365077-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408690 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408690 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408690 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408816 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408761    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.408867 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.409011 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.408988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.410763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.410743    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.411042 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.411024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014e29d1-f292-4299-8a8b-ea492b365077-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.416362 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.416342    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m29m\" (UniqueName: \"kubernetes.io/projected/014e29d1-f292-4299-8a8b-ea492b365077-kube-api-access-2m29m\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.452433 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.452391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:18.733361 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:18.733332    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"]
Sep 18 22:22:18.735258 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:22:18.735228    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014e29d1_f292_4299_8a8b_ea492b365077.slice/crio-5e57de967eff9b9b06846dd7f05059d18123f9172a2a235cbf7219ea95b1f531 WatchSource:0}: Error finding container 5e57de967eff9b9b06846dd7f05059d18123f9172a2a235cbf7219ea95b1f531: Status 404 returned error can't find the container with id 5e57de967eff9b9b06846dd7f05059d18123f9172a2a235cbf7219ea95b1f531
Sep 18 22:22:19.377328 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:19.377294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" event={"ID":"014e29d1-f292-4299-8a8b-ea492b365077","Type":"ContainerStarted","Data":"7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d"}
Sep 18 22:22:19.377708 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:19.377346    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" event={"ID":"014e29d1-f292-4299-8a8b-ea492b365077","Type":"ContainerStarted","Data":"5e57de967eff9b9b06846dd7f05059d18123f9172a2a235cbf7219ea95b1f531"}
Sep 18 22:22:19.398556 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:19.398499    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" podStartSLOduration=1.398483768 podStartE2EDuration="1.398483768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:19.397493578 +0000 UTC m=+1896.430672473" watchObservedRunningTime="2026-09-18 22:22:19.398483768 +0000 UTC m=+1896.431662709"
Sep 18 22:22:19.452878 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:19.452839    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:26.123177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.123143    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"]
Sep 18 22:22:26.130203 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.130173    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"]
Sep 18 22:22:26.130461 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.130420    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd" podUID="a2cb57c8-be39-4412-aa05-8ea9082556d8" containerName="istio-proxy" containerID="cri-o://f84e54b3962a1a0514d9be084f89c42cb9f817aa6ca7aea5073fe9d7db33c117" gracePeriod=30
Sep 18 22:22:26.134063 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:22:26.134024    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:22:26.134063 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:22:26.134063 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:22:26.134063 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d" cmd=["/bin/sleep","15"]
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.134087    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" message="" err=<
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:22:26.134104    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" podUID="014e29d1-f292-4299-8a8b-ea492b365077" containerName="main" containerID="cri-o://7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d"
Sep 18 22:22:26.134320 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.134188    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" podUID="014e29d1-f292-4299-8a8b-ea492b365077" containerName="main" containerID="cri-o://7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d" gracePeriod=60
Sep 18 22:22:26.393587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.393559    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:26.436564 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.436528    2582 generic.go:356] "Generic (PLEG): container finished" podID="014e29d1-f292-4299-8a8b-ea492b365077" containerID="7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d" exitCode=0
Sep 18 22:22:26.436740 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.436581    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"
Sep 18 22:22:26.436740 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.436610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" event={"ID":"014e29d1-f292-4299-8a8b-ea492b365077","Type":"ContainerDied","Data":"7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d"}
Sep 18 22:22:26.436740 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.436695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n" event={"ID":"014e29d1-f292-4299-8a8b-ea492b365077","Type":"ContainerDied","Data":"5e57de967eff9b9b06846dd7f05059d18123f9172a2a235cbf7219ea95b1f531"}
Sep 18 22:22:26.436861 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.436726    2582 scope.go:117] "RemoveContainer" containerID="7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d"
Sep 18 22:22:26.451003 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.450978    2582 scope.go:117] "RemoveContainer" containerID="7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d"
Sep 18 22:22:26.451336 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:22:26.451311    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d\": container with ID starting with 7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d not found: ID does not exist" containerID="7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d"
Sep 18 22:22:26.451420 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.451350    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d"} err="failed to get container status \"7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d\": rpc error: code = NotFound desc = could not find container \"7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d\": container with ID starting with 7522a48bbd84146c263dcf16f6347c799d5e2dca07a2f060dc0f1eb32ab4352d not found: ID does not exist"
Sep 18 22:22:26.472778 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.472744    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-model-cache\") pod \"014e29d1-f292-4299-8a8b-ea492b365077\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") "
Sep 18 22:22:26.472965 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.472798    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-home\") pod \"014e29d1-f292-4299-8a8b-ea492b365077\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") "
Sep 18 22:22:26.472965 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.472832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m29m\" (UniqueName: \"kubernetes.io/projected/014e29d1-f292-4299-8a8b-ea492b365077-kube-api-access-2m29m\") pod \"014e29d1-f292-4299-8a8b-ea492b365077\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") "
Sep 18 22:22:26.472965 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.472897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-tmp-dir\") pod \"014e29d1-f292-4299-8a8b-ea492b365077\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") "
Sep 18 22:22:26.473115 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.472986    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014e29d1-f292-4299-8a8b-ea492b365077-tls-certs\") pod \"014e29d1-f292-4299-8a8b-ea492b365077\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") "
Sep 18 22:22:26.473115 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.473068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-home" (OuterVolumeSpecName: "home") pod "014e29d1-f292-4299-8a8b-ea492b365077" (UID: "014e29d1-f292-4299-8a8b-ea492b365077"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:26.473115 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.473081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-model-cache" (OuterVolumeSpecName: "model-cache") pod "014e29d1-f292-4299-8a8b-ea492b365077" (UID: "014e29d1-f292-4299-8a8b-ea492b365077"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:26.473115 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.473093    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-dshm\") pod \"014e29d1-f292-4299-8a8b-ea492b365077\" (UID: \"014e29d1-f292-4299-8a8b-ea492b365077\") "
Sep 18 22:22:26.473298 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.473243    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "014e29d1-f292-4299-8a8b-ea492b365077" (UID: "014e29d1-f292-4299-8a8b-ea492b365077"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:26.473451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.473426    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:26.473451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.473453    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:26.473638 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.473465    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:26.475319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.475293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014e29d1-f292-4299-8a8b-ea492b365077-kube-api-access-2m29m" (OuterVolumeSpecName: "kube-api-access-2m29m") pod "014e29d1-f292-4299-8a8b-ea492b365077" (UID: "014e29d1-f292-4299-8a8b-ea492b365077"). InnerVolumeSpecName "kube-api-access-2m29m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:22:26.475451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.475425    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014e29d1-f292-4299-8a8b-ea492b365077-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "014e29d1-f292-4299-8a8b-ea492b365077" (UID: "014e29d1-f292-4299-8a8b-ea492b365077"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:22:26.475519 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.475468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-dshm" (OuterVolumeSpecName: "dshm") pod "014e29d1-f292-4299-8a8b-ea492b365077" (UID: "014e29d1-f292-4299-8a8b-ea492b365077"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:26.574005 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.573960    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m29m\" (UniqueName: \"kubernetes.io/projected/014e29d1-f292-4299-8a8b-ea492b365077-kube-api-access-2m29m\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:26.574005 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.573993    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014e29d1-f292-4299-8a8b-ea492b365077-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:26.574005 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.574003    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014e29d1-f292-4299-8a8b-ea492b365077-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:26.762137 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.762108    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"]
Sep 18 22:22:26.767746 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.767714    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-5tn9n"]
Sep 18 22:22:26.974140 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:26.974105    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014e29d1-f292-4299-8a8b-ea492b365077" path="/var/lib/kubelet/pods/014e29d1-f292-4299-8a8b-ea492b365077/volumes"
Sep 18 22:22:28.297139 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.296475    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd" podUID="a2cb57c8-be39-4412-aa05-8ea9082556d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": read tcp 10.133.0.2:59586->10.133.0.45:15021: read: connection reset by peer"
Sep 18 22:22:28.460574 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.460543    2582 generic.go:356] "Generic (PLEG): container finished" podID="a2cb57c8-be39-4412-aa05-8ea9082556d8" containerID="f84e54b3962a1a0514d9be084f89c42cb9f817aa6ca7aea5073fe9d7db33c117" exitCode=0
Sep 18 22:22:28.460704 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.460642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd" event={"ID":"a2cb57c8-be39-4412-aa05-8ea9082556d8","Type":"ContainerDied","Data":"f84e54b3962a1a0514d9be084f89c42cb9f817aa6ca7aea5073fe9d7db33c117"}
Sep 18 22:22:28.474886 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.474862    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:22:28.487477 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487453    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-envoy\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487612 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-token\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487612 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487515    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-data\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487612 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-socket\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487612 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487589    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv4sr\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-kube-api-access-dv4sr\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487828 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-certs\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487828 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487647    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cb57c8-be39-4412-aa05-8ea9082556d8-istiod-ca-cert\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487828 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-credential-socket\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.487828 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487740    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-podinfo\") pod \"a2cb57c8-be39-4412-aa05-8ea9082556d8\" (UID: \"a2cb57c8-be39-4412-aa05-8ea9082556d8\") "
Sep 18 22:22:28.488066 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487833    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:28.488066 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.487933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:28.488176 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.488077    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.488176 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.488097    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-data\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.488282 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.488180    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cb57c8-be39-4412-aa05-8ea9082556d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:22:28.488340 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.488318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:28.488396 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.488379    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:28.489869 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.489847    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:22:28.490149 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.490131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-kube-api-access-dv4sr" (OuterVolumeSpecName: "kube-api-access-dv4sr") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "kube-api-access-dv4sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:22:28.490228 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.490187    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:28.490228 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.490209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2cb57c8-be39-4412-aa05-8ea9082556d8" (UID: "a2cb57c8-be39-4412-aa05-8ea9082556d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:22:28.588499 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.588464    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv4sr\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-kube-api-access-dv4sr\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.588499 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.588492    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cb57c8-be39-4412-aa05-8ea9082556d8-istiod-ca-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.588499 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.588505    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-credential-socket\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.588719 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.588513    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-podinfo\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.588719 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.588522    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-envoy\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.588719 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.588530    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cb57c8-be39-4412-aa05-8ea9082556d8-istio-token\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:28.588719 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:28.588539    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cb57c8-be39-4412-aa05-8ea9082556d8-workload-socket\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:29.467472 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:29.467432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd" event={"ID":"a2cb57c8-be39-4412-aa05-8ea9082556d8","Type":"ContainerDied","Data":"0b995b149b4afea03b10370cbc7146a88308e60850ff12633befb48386a30f0d"}
Sep 18 22:22:29.467472 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:29.467472    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"
Sep 18 22:22:29.467992 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:29.467500    2582 scope.go:117] "RemoveContainer" containerID="f84e54b3962a1a0514d9be084f89c42cb9f817aa6ca7aea5073fe9d7db33c117"
Sep 18 22:22:29.485812 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:29.485783    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"]
Sep 18 22:22:29.489555 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:29.489530    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-6xcpd"]
Sep 18 22:22:30.974855 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:30.974819    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2cb57c8-be39-4412-aa05-8ea9082556d8" path="/var/lib/kubelet/pods/a2cb57c8-be39-4412-aa05-8ea9082556d8/volumes"
Sep 18 22:22:40.069673 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.069635    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"]
Sep 18 22:22:40.070059 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.070045    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014e29d1-f292-4299-8a8b-ea492b365077" containerName="main"
Sep 18 22:22:40.070115 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.070061    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="014e29d1-f292-4299-8a8b-ea492b365077" containerName="main"
Sep 18 22:22:40.070115 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.070090    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2cb57c8-be39-4412-aa05-8ea9082556d8" containerName="istio-proxy"
Sep 18 22:22:40.070115 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.070098    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2cb57c8-be39-4412-aa05-8ea9082556d8" containerName="istio-proxy"
Sep 18 22:22:40.070210 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.070152    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2cb57c8-be39-4412-aa05-8ea9082556d8" containerName="istio-proxy"
Sep 18 22:22:40.070210 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.070159    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="014e29d1-f292-4299-8a8b-ea492b365077" containerName="main"
Sep 18 22:22:40.073229 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.073208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.087701 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.087672    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 22:22:40.087885 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.087727    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-8rzsp\""
Sep 18 22:22:40.088050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.088014    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 22:22:40.088240 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.088164    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 18 22:22:40.091611 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.091588    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"]
Sep 18 22:22:40.189168 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.189132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.189352 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.189181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.189352 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.189203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f183d488-bbe0-4802-919f-76411e5a87ab-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.189352 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.189253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.189352 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.189271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.189352 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.189320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbh6\" (UniqueName: \"kubernetes.io/projected/f183d488-bbe0-4802-919f-76411e5a87ab-kube-api-access-bjbh6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290199 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbh6\" (UniqueName: \"kubernetes.io/projected/f183d488-bbe0-4802-919f-76411e5a87ab-kube-api-access-bjbh6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f183d488-bbe0-4802-919f-76411e5a87ab-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290691 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290668    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290756 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.290756 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.290734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.292642 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.292620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.293004 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.292984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f183d488-bbe0-4802-919f-76411e5a87ab-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.301319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.301299    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbh6\" (UniqueName: \"kubernetes.io/projected/f183d488-bbe0-4802-919f-76411e5a87ab-kube-api-access-bjbh6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.391397 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.391365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:40.665740 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:40.665715    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"]
Sep 18 22:22:40.667561 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:22:40.667537    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf183d488_bbe0_4802_919f_76411e5a87ab.slice/crio-9ea3370d21ee4491a4a6d12e5b136cd6c03bffedbf2ec686f4a6b211acf40a12 WatchSource:0}: Error finding container 9ea3370d21ee4491a4a6d12e5b136cd6c03bffedbf2ec686f4a6b211acf40a12: Status 404 returned error can't find the container with id 9ea3370d21ee4491a4a6d12e5b136cd6c03bffedbf2ec686f4a6b211acf40a12
Sep 18 22:22:41.534005 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:41.533965    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" event={"ID":"f183d488-bbe0-4802-919f-76411e5a87ab","Type":"ContainerStarted","Data":"688ca5232d5ed21af93b6008c80af8791036f853d8a2177008d0f3d1db8a90fd"}
Sep 18 22:22:41.534375 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:41.534023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" event={"ID":"f183d488-bbe0-4802-919f-76411e5a87ab","Type":"ContainerStarted","Data":"9ea3370d21ee4491a4a6d12e5b136cd6c03bffedbf2ec686f4a6b211acf40a12"}
Sep 18 22:22:41.553563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:41.553509    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" podStartSLOduration=1.5534925670000002 podStartE2EDuration="1.553492567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:41.551547803 +0000 UTC m=+1918.584726698" watchObservedRunningTime="2026-09-18 22:22:41.553492567 +0000 UTC m=+1918.586671461"
Sep 18 22:22:42.392031 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:42.392002    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:45.382839 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.382806    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"]
Sep 18 22:22:45.391742 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:22:45.391708    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:22:45.391742 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:22:45.391742 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:22:45.391742 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="688ca5232d5ed21af93b6008c80af8791036f853d8a2177008d0f3d1db8a90fd" cmd=["/bin/sleep","15"]
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.391758    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" message="" err=<
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:22:45.391771    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" podUID="f183d488-bbe0-4802-919f-76411e5a87ab" containerName="main" containerID="cri-o://688ca5232d5ed21af93b6008c80af8791036f853d8a2177008d0f3d1db8a90fd"
Sep 18 22:22:45.392009 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.391816    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" podUID="f183d488-bbe0-4802-919f-76411e5a87ab" containerName="main" containerID="cri-o://688ca5232d5ed21af93b6008c80af8791036f853d8a2177008d0f3d1db8a90fd" gracePeriod=60
Sep 18 22:22:45.557024 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.556994    2582 generic.go:356] "Generic (PLEG): container finished" podID="f183d488-bbe0-4802-919f-76411e5a87ab" containerID="688ca5232d5ed21af93b6008c80af8791036f853d8a2177008d0f3d1db8a90fd" exitCode=0
Sep 18 22:22:45.557157 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.557055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" event={"ID":"f183d488-bbe0-4802-919f-76411e5a87ab","Type":"ContainerDied","Data":"688ca5232d5ed21af93b6008c80af8791036f853d8a2177008d0f3d1db8a90fd"}
Sep 18 22:22:45.637200 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.637137    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:45.739742 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.739710    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-dshm\") pod \"f183d488-bbe0-4802-919f-76411e5a87ab\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") "
Sep 18 22:22:45.739925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.739760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-model-cache\") pod \"f183d488-bbe0-4802-919f-76411e5a87ab\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") "
Sep 18 22:22:45.739925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.739818    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f183d488-bbe0-4802-919f-76411e5a87ab-tls-certs\") pod \"f183d488-bbe0-4802-919f-76411e5a87ab\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") "
Sep 18 22:22:45.739925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.739835    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-home\") pod \"f183d488-bbe0-4802-919f-76411e5a87ab\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") "
Sep 18 22:22:45.739925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.739850    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-tmp-dir\") pod \"f183d488-bbe0-4802-919f-76411e5a87ab\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") "
Sep 18 22:22:45.739925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.739889    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjbh6\" (UniqueName: \"kubernetes.io/projected/f183d488-bbe0-4802-919f-76411e5a87ab-kube-api-access-bjbh6\") pod \"f183d488-bbe0-4802-919f-76411e5a87ab\" (UID: \"f183d488-bbe0-4802-919f-76411e5a87ab\") "
Sep 18 22:22:45.740218 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.740101    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-home" (OuterVolumeSpecName: "home") pod "f183d488-bbe0-4802-919f-76411e5a87ab" (UID: "f183d488-bbe0-4802-919f-76411e5a87ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:45.740218 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.740110    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "f183d488-bbe0-4802-919f-76411e5a87ab" (UID: "f183d488-bbe0-4802-919f-76411e5a87ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:45.740218 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.740177    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f183d488-bbe0-4802-919f-76411e5a87ab" (UID: "f183d488-bbe0-4802-919f-76411e5a87ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:45.742061 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.742027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-dshm" (OuterVolumeSpecName: "dshm") pod "f183d488-bbe0-4802-919f-76411e5a87ab" (UID: "f183d488-bbe0-4802-919f-76411e5a87ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:22:45.742161 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.742096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f183d488-bbe0-4802-919f-76411e5a87ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f183d488-bbe0-4802-919f-76411e5a87ab" (UID: "f183d488-bbe0-4802-919f-76411e5a87ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:22:45.742161 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.742103    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f183d488-bbe0-4802-919f-76411e5a87ab-kube-api-access-bjbh6" (OuterVolumeSpecName: "kube-api-access-bjbh6") pod "f183d488-bbe0-4802-919f-76411e5a87ab" (UID: "f183d488-bbe0-4802-919f-76411e5a87ab"). InnerVolumeSpecName "kube-api-access-bjbh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:22:45.840605 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.840567    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbh6\" (UniqueName: \"kubernetes.io/projected/f183d488-bbe0-4802-919f-76411e5a87ab-kube-api-access-bjbh6\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:45.840605 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.840598    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:45.840605 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.840608    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:45.840806 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.840618    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f183d488-bbe0-4802-919f-76411e5a87ab-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:45.840806 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.840625    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:45.840806 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:45.840652    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f183d488-bbe0-4802-919f-76411e5a87ab-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:22:46.563888 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:46.563853    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"
Sep 18 22:22:46.564324 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:46.563894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz" event={"ID":"f183d488-bbe0-4802-919f-76411e5a87ab","Type":"ContainerDied","Data":"9ea3370d21ee4491a4a6d12e5b136cd6c03bffedbf2ec686f4a6b211acf40a12"}
Sep 18 22:22:46.564324 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:46.563979    2582 scope.go:117] "RemoveContainer" containerID="688ca5232d5ed21af93b6008c80af8791036f853d8a2177008d0f3d1db8a90fd"
Sep 18 22:22:46.585606 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:46.585579    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"]
Sep 18 22:22:46.589642 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:46.589618    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvjzzz"]
Sep 18 22:22:46.974131 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:22:46.974090    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f183d488-bbe0-4802-919f-76411e5a87ab" path="/var/lib/kubelet/pods/f183d488-bbe0-4802-919f-76411e5a87ab/volumes"
Sep 18 22:23:23.325345 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.325306    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"]
Sep 18 22:23:23.325789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.325708    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f183d488-bbe0-4802-919f-76411e5a87ab" containerName="main"
Sep 18 22:23:23.325789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.325725    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f183d488-bbe0-4802-919f-76411e5a87ab" containerName="main"
Sep 18 22:23:23.325869 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.325827    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f183d488-bbe0-4802-919f-76411e5a87ab" containerName="main"
Sep 18 22:23:23.329121 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.329104    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.331664 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.331607    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:23:23.331792 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.331687    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 18 22:23:23.331792 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.331743    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 18 22:23:23.331926 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.331887    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-q5sc9\""
Sep 18 22:23:23.332048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.332031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 18 22:23:23.332097 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.332041    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 18 22:23:23.358670 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kserve-provision-location\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.358811 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tls-certs\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.358811 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff75f63f-3cc7-4119-97e8-cdb819eb7953-cabundle-cert\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.358811 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffq2x\" (UniqueName: \"kubernetes.io/projected/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kube-api-access-ffq2x\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.358993 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-model-cache\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.358993 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tmp-dir\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.358993 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-home\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.358993 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.358934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-dshm\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.397678 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.397644    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"]
Sep 18 22:23:23.460177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-model-cache\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460300 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tmp-dir\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460300 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-home\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460300 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-dshm\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460452 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kserve-provision-location\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460452 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tls-certs\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460613 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff75f63f-3cc7-4119-97e8-cdb819eb7953-cabundle-cert\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460613 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffq2x\" (UniqueName: \"kubernetes.io/projected/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kube-api-access-ffq2x\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460613 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-model-cache\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460811 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tmp-dir\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-home\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.460904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.460883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kserve-provision-location\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.461183 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.461162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff75f63f-3cc7-4119-97e8-cdb819eb7953-cabundle-cert\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.462543 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.462521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-dshm\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.462959 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.462929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tls-certs\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.469054 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.469036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffq2x\" (UniqueName: \"kubernetes.io/projected/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kube-api-access-ffq2x\") pod \"auth-enabled-test-kserve-98db79b4-46mjc\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.640941 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.640906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:23.926172 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:23.926144    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"]
Sep 18 22:23:23.927346 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:23:23.927316    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff75f63f_3cc7_4119_97e8_cdb819eb7953.slice/crio-a7c21b3891e445793ebc2f2733c4ae9b196f37a0a0d68b8f666fd0c3ebf5649d WatchSource:0}: Error finding container a7c21b3891e445793ebc2f2733c4ae9b196f37a0a0d68b8f666fd0c3ebf5649d: Status 404 returned error can't find the container with id a7c21b3891e445793ebc2f2733c4ae9b196f37a0a0d68b8f666fd0c3ebf5649d
Sep 18 22:23:24.763012 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:24.762970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" event={"ID":"ff75f63f-3cc7-4119-97e8-cdb819eb7953","Type":"ContainerStarted","Data":"46d6f25b642146d4aa3a9690ca4ae203b0935a87ee12b267c6e2524536bed96b"}
Sep 18 22:23:24.763494 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:24.763027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" event={"ID":"ff75f63f-3cc7-4119-97e8-cdb819eb7953","Type":"ContainerStarted","Data":"a7c21b3891e445793ebc2f2733c4ae9b196f37a0a0d68b8f666fd0c3ebf5649d"}
Sep 18 22:23:26.777426 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:26.777393    2582 generic.go:356] "Generic (PLEG): container finished" podID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerID="46d6f25b642146d4aa3a9690ca4ae203b0935a87ee12b267c6e2524536bed96b" exitCode=0
Sep 18 22:23:26.777835 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:26.777464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" event={"ID":"ff75f63f-3cc7-4119-97e8-cdb819eb7953","Type":"ContainerDied","Data":"46d6f25b642146d4aa3a9690ca4ae203b0935a87ee12b267c6e2524536bed96b"}
Sep 18 22:23:29.102719 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:29.102610    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9_e6c236b5-5165-46bf-a4b0-a9cbd13ca414/llm-d-routing-sidecar/0.log"
Sep 18 22:23:29.111993 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:29.111918    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-9c386aa9_scheduler-v06-pd-migration-test-kserve-5c479b44c4-r77z9_e6c236b5-5165-46bf-a4b0-a9cbd13ca414/main/0.log"
Sep 18 22:23:51.073242 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.073192    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"]
Sep 18 22:23:51.153206 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.153169    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"]
Sep 18 22:23:51.153385 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.153309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.157379 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.157351    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-nwkkw\""
Sep 18 22:23:51.157379 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.157374    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 22:23:51.157606 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.157395    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-7kq6f\""
Sep 18 22:23:51.157606 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.157441    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 22:23:51.157726 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.157709    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 22:23:51.232122 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.232086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.232325 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.232140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.232325 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.232175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hld\" (UniqueName: \"kubernetes.io/projected/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-kube-api-access-v7hld\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.232455 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.232331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.232455 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.232413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.232550 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.232462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333418 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333418 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333418 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333689 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333689 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333689 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hld\" (UniqueName: \"kubernetes.io/projected/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-kube-api-access-v7hld\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333838 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.333891 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.333851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.334030 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.334007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.336177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.336154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.336355 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.336334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.350103 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.350077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hld\" (UniqueName: \"kubernetes.io/projected/0341eae0-7d5e-4ccd-a01c-846a7dccfb2e-kube-api-access-v7hld\") pod \"scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7\" (UID: \"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:51.464590 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:51.464548    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:54.264148 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:54.264044    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"]
Sep 18 22:23:54.266371 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:23:54.266333    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0341eae0_7d5e_4ccd_a01c_846a7dccfb2e.slice/crio-72038cb96052ea2114d60d9d6ea9bb11edcf9db39da4baca6db448c6aa501c4f WatchSource:0}: Error finding container 72038cb96052ea2114d60d9d6ea9bb11edcf9db39da4baca6db448c6aa501c4f: Status 404 returned error can't find the container with id 72038cb96052ea2114d60d9d6ea9bb11edcf9db39da4baca6db448c6aa501c4f
Sep 18 22:23:54.973711 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:54.973681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" event={"ID":"ff75f63f-3cc7-4119-97e8-cdb819eb7953","Type":"ContainerStarted","Data":"0fdf72b1c55b3e368535efcad55126d7829f32a64ef5ce76dab439e58a205023"}
Sep 18 22:23:54.973808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:54.973729    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7" event={"ID":"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e","Type":"ContainerStarted","Data":"5510762863da75f829979aeafd6165a2e0586fd1d8462c716016927089bd95b5"}
Sep 18 22:23:54.973808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:54.973754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7" event={"ID":"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e","Type":"ContainerStarted","Data":"72038cb96052ea2114d60d9d6ea9bb11edcf9db39da4baca6db448c6aa501c4f"}
Sep 18 22:23:54.973808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:54.973783    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:23:55.028687 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:55.028635    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podStartSLOduration=2.425691171 podStartE2EDuration="32.028618448s" totalImagesPullingTime="27.413939959s" totalInitContainerRuntime="2.188987318s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:23:55.025585127 +0000 UTC m=+1992.058764025" watchObservedRunningTime="2026-09-18 22:23:55.028618448 +0000 UTC m=+1992.061797382"
Sep 18 22:23:55.641923 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:55.641885    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:23:55.984265 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:55.984153    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7" event={"ID":"0341eae0-7d5e-4ccd-a01c-846a7dccfb2e","Type":"ContainerStarted","Data":"10a20967a7a53613d1634ca2a4722220f651e896f39ce7b7618ade8924100324"}
Sep 18 22:23:56.007399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:56.007345    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7" podStartSLOduration=5.007330343 podStartE2EDuration="5.007330343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:23:56.004921591 +0000 UTC m=+1993.038100490" watchObservedRunningTime="2026-09-18 22:23:56.007330343 +0000 UTC m=+1993.040509237"
Sep 18 22:23:56.465286 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:23:56.465250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:24:01.465196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:01.465152    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:24:01.477580 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:01.477555    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:24:02.030137 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:02.030105    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:24:03.641902 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:03.641853    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:24:03.643559 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:03.643530    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:24:13.039805 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:13.039769    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7"
Sep 18 22:24:13.642133 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:13.642092    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:24:23.642070 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:23.642018    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:24:33.642203 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:33.642161    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:24:43.641471 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:43.641426    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:24:53.641888 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:24:53.641841    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:25:03.641923 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:03.641875    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:25:13.641329 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:13.641286    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:25:23.641643 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:23.641601    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:25:33.641786 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:33.641733    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:25:43.641900 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:43.641840    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:25:45.090520 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:45.090494    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:25:45.096604 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:45.096576    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:25:45.099493 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:45.099472    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:25:45.105722 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:45.105703    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:25:53.653796 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:53.653760    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:25:53.661564 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:53.661535    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:25:59.068635 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:25:59.068602    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"]
Sep 18 22:26:14.090880 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.090834    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" containerID="cri-o://0fdf72b1c55b3e368535efcad55126d7829f32a64ef5ce76dab439e58a205023" gracePeriod=45
Sep 18 22:26:14.654991 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.654925    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 18 22:26:14.767093 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.767059    2582 generic.go:356] "Generic (PLEG): container finished" podID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerID="0fdf72b1c55b3e368535efcad55126d7829f32a64ef5ce76dab439e58a205023" exitCode=0
Sep 18 22:26:14.767253 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.767129    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" event={"ID":"ff75f63f-3cc7-4119-97e8-cdb819eb7953","Type":"ContainerDied","Data":"0fdf72b1c55b3e368535efcad55126d7829f32a64ef5ce76dab439e58a205023"}
Sep 18 22:26:14.832862 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.832838    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:26:14.893246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893210    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tmp-dir\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893252    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tls-certs\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893504 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893269    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff75f63f-3cc7-4119-97e8-cdb819eb7953-cabundle-cert\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893504 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-home\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893504 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893311    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffq2x\" (UniqueName: \"kubernetes.io/projected/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kube-api-access-ffq2x\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893504 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893342    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kserve-provision-location\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893504 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893366    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-model-cache\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893504 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893387    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-dshm\") pod \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\" (UID: \"ff75f63f-3cc7-4119-97e8-cdb819eb7953\") "
Sep 18 22:26:14.893835 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893798    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff75f63f-3cc7-4119-97e8-cdb819eb7953-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:26:14.894002 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.893939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-home" (OuterVolumeSpecName: "home") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:26:14.894238 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.894125    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:26:14.895844 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.895817    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:26:14.896000 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.895932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-dshm" (OuterVolumeSpecName: "dshm") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:26:14.896181 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.896161    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kube-api-access-ffq2x" (OuterVolumeSpecName: "kube-api-access-ffq2x") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "kube-api-access-ffq2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:26:14.911504 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.911457    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:26:14.915427 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.915404    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff75f63f-3cc7-4119-97e8-cdb819eb7953" (UID: "ff75f63f-3cc7-4119-97e8-cdb819eb7953"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:26:14.994136 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994111    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:14.994136 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994135    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff75f63f-3cc7-4119-97e8-cdb819eb7953-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:14.994274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994143    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff75f63f-3cc7-4119-97e8-cdb819eb7953-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:14.994274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994153    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:14.994274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994161    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffq2x\" (UniqueName: \"kubernetes.io/projected/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kube-api-access-ffq2x\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:14.994274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994168    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:14.994274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994177    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:14.994274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:14.994186    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff75f63f-3cc7-4119-97e8-cdb819eb7953-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:26:15.773857 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:15.773820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc" event={"ID":"ff75f63f-3cc7-4119-97e8-cdb819eb7953","Type":"ContainerDied","Data":"a7c21b3891e445793ebc2f2733c4ae9b196f37a0a0d68b8f666fd0c3ebf5649d"}
Sep 18 22:26:15.773857 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:15.773851    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"
Sep 18 22:26:15.774466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:15.773883    2582 scope.go:117] "RemoveContainer" containerID="0fdf72b1c55b3e368535efcad55126d7829f32a64ef5ce76dab439e58a205023"
Sep 18 22:26:15.783319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:15.783299    2582 scope.go:117] "RemoveContainer" containerID="46d6f25b642146d4aa3a9690ca4ae203b0935a87ee12b267c6e2524536bed96b"
Sep 18 22:26:15.794282 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:15.794258    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"]
Sep 18 22:26:15.798228 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:15.798206    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-98db79b4-46mjc"]
Sep 18 22:26:16.975160 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:16.975123    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" path="/var/lib/kubelet/pods/ff75f63f-3cc7-4119-97e8-cdb819eb7953/volumes"
Sep 18 22:26:51.625239 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.625204    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"]
Sep 18 22:26:51.625696 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.625576    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="storage-initializer"
Sep 18 22:26:51.625696 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.625587    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="storage-initializer"
Sep 18 22:26:51.625696 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.625596    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main"
Sep 18 22:26:51.625696 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.625601    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main"
Sep 18 22:26:51.625696 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.625654    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff75f63f-3cc7-4119-97e8-cdb819eb7953" containerName="main"
Sep 18 22:26:51.628786 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.628766    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.633325 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.633304    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 18 22:26:51.633471 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.633307    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 22:26:51.633567 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.633483    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 22:26:51.633567 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.633539    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:26:51.633688 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.633565    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 22:26:51.633688 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.633572    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-lwbg7\""
Sep 18 22:26:51.643606 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.643584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"]
Sep 18 22:26:51.711253 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-home\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.711253 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a26de2-4c45-4ec2-94a7-566d3f810667-cabundle-cert\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.711445 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp72h\" (UniqueName: \"kubernetes.io/projected/c6a26de2-4c45-4ec2-94a7-566d3f810667-kube-api-access-tp72h\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.711445 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-model-cache\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.711445 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-tmp-dir\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.711445 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-dshm\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.711574 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a26de2-4c45-4ec2-94a7-566d3f810667-tls-certs\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.711574 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.711471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.812898 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.812861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a26de2-4c45-4ec2-94a7-566d3f810667-tls-certs\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.812898 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.812898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813143 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.812963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-home\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813143 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.812993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a26de2-4c45-4ec2-94a7-566d3f810667-cabundle-cert\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813143 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp72h\" (UniqueName: \"kubernetes.io/projected/c6a26de2-4c45-4ec2-94a7-566d3f810667-kube-api-access-tp72h\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813143 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-model-cache\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813143 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-tmp-dir\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813402 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-dshm\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813402 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-home\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813517 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-model-cache\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813578 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-tmp-dir\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813637 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.813829 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.813810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a26de2-4c45-4ec2-94a7-566d3f810667-cabundle-cert\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.815555 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.815527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-dshm\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.815644 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.815582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a26de2-4c45-4ec2-94a7-566d3f810667-tls-certs\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.821495 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.821474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp72h\" (UniqueName: \"kubernetes.io/projected/c6a26de2-4c45-4ec2-94a7-566d3f810667-kube-api-access-tp72h\") pod \"auth-invalid-token-test-kserve-56446fbdbd-4hjj6\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:51.939352 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:51.939260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:26:52.249796 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:52.249658    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"]
Sep 18 22:26:52.252603 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:26:52.252574    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a26de2_4c45_4ec2_94a7_566d3f810667.slice/crio-81c5dcd2c56e33055929bacda636a2cbb6c12fecd4031cb77396ca582dab458b WatchSource:0}: Error finding container 81c5dcd2c56e33055929bacda636a2cbb6c12fecd4031cb77396ca582dab458b: Status 404 returned error can't find the container with id 81c5dcd2c56e33055929bacda636a2cbb6c12fecd4031cb77396ca582dab458b
Sep 18 22:26:52.975900 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:52.975823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" event={"ID":"c6a26de2-4c45-4ec2-94a7-566d3f810667","Type":"ContainerStarted","Data":"e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285"}
Sep 18 22:26:52.975900 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:52.975873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" event={"ID":"c6a26de2-4c45-4ec2-94a7-566d3f810667","Type":"ContainerStarted","Data":"81c5dcd2c56e33055929bacda636a2cbb6c12fecd4031cb77396ca582dab458b"}
Sep 18 22:26:54.986535 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:54.986503    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerID="e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285" exitCode=0
Sep 18 22:26:54.986897 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:54.986575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" event={"ID":"c6a26de2-4c45-4ec2-94a7-566d3f810667","Type":"ContainerDied","Data":"e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285"}
Sep 18 22:26:55.994732 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:55.994692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" event={"ID":"c6a26de2-4c45-4ec2-94a7-566d3f810667","Type":"ContainerStarted","Data":"4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d"}
Sep 18 22:26:56.021002 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:56.020917    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podStartSLOduration=2.8424045959999997 podStartE2EDuration="5.020896071s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.178491475s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:26:56.018387023 +0000 UTC m=+2173.051565919" watchObservedRunningTime="2026-09-18 22:26:56.020896071 +0000 UTC m=+2173.054074966"
Sep 18 22:26:56.940305 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:26:56.940265    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:27:01.939538 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:27:01.939491    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:27:01.941155 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:27:01.941125    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:27:11.940484 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:27:11.940438    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:27:21.939907 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:27:21.939860    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:27:31.939662 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:27:31.939617    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:27:41.940622 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:27:41.940570    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:27:51.940041 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:27:51.939996    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:28:01.940243 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:01.940145    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:28:11.940263 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:11.940212    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:28:21.939862 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:21.939822    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:28:31.940574 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:31.940525    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:28:41.939851 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:41.939804    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:28:51.950428 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:51.950396    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:28:51.958471 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:51.958445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:28:53.832916 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:28:53.832877    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"]
Sep 18 22:29:08.860586 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:08.860539    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" containerID="cri-o://4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d" gracePeriod=45
Sep 18 22:29:08.951650 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:08.951604    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 18 22:29:09.613673 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.613648    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:29:09.748477 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-kserve-provision-location\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.748477 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748439    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-home\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.748477 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748479    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-model-cache\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.748769 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp72h\" (UniqueName: \"kubernetes.io/projected/c6a26de2-4c45-4ec2-94a7-566d3f810667-kube-api-access-tp72h\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.748769 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748530    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-tmp-dir\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.748769 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748564    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-dshm\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.748769 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748608    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a26de2-4c45-4ec2-94a7-566d3f810667-cabundle-cert\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.748769 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a26de2-4c45-4ec2-94a7-566d3f810667-tls-certs\") pod \"c6a26de2-4c45-4ec2-94a7-566d3f810667\" (UID: \"c6a26de2-4c45-4ec2-94a7-566d3f810667\") "
Sep 18 22:29:09.749060 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748789    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:29:09.749060 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.748935    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:09.749344 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.749209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-home" (OuterVolumeSpecName: "home") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:29:09.749344 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.749335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a26de2-4c45-4ec2-94a7-566d3f810667-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:29:09.751228 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.751142    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a26de2-4c45-4ec2-94a7-566d3f810667-kube-api-access-tp72h" (OuterVolumeSpecName: "kube-api-access-tp72h") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "kube-api-access-tp72h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:29:09.751450 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.751429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a26de2-4c45-4ec2-94a7-566d3f810667-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:29:09.751613 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.751593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-dshm" (OuterVolumeSpecName: "dshm") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:29:09.758702 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.758673    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerID="4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d" exitCode=0
Sep 18 22:29:09.758795 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.758728    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"
Sep 18 22:29:09.758795 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.758750    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" event={"ID":"c6a26de2-4c45-4ec2-94a7-566d3f810667","Type":"ContainerDied","Data":"4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d"}
Sep 18 22:29:09.758870 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.758812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6" event={"ID":"c6a26de2-4c45-4ec2-94a7-566d3f810667","Type":"ContainerDied","Data":"81c5dcd2c56e33055929bacda636a2cbb6c12fecd4031cb77396ca582dab458b"}
Sep 18 22:29:09.758870 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.758844    2582 scope.go:117] "RemoveContainer" containerID="4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d"
Sep 18 22:29:09.763742 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.763716    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:29:09.778403 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.778378    2582 scope.go:117] "RemoveContainer" containerID="e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285"
Sep 18 22:29:09.779922 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.779886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6a26de2-4c45-4ec2-94a7-566d3f810667" (UID: "c6a26de2-4c45-4ec2-94a7-566d3f810667"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:29:09.788782 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.788764    2582 scope.go:117] "RemoveContainer" containerID="4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d"
Sep 18 22:29:09.789118 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:29:09.789094    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d\": container with ID starting with 4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d not found: ID does not exist" containerID="4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d"
Sep 18 22:29:09.789238 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.789126    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d"} err="failed to get container status \"4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d\": rpc error: code = NotFound desc = could not find container \"4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d\": container with ID starting with 4a17a1769e64eaba4eaf7461db086b1970ce0c765eee67bcf3a7a318b4b01c8d not found: ID does not exist"
Sep 18 22:29:09.789238 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.789152    2582 scope.go:117] "RemoveContainer" containerID="e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285"
Sep 18 22:29:09.789447 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:29:09.789428    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285\": container with ID starting with e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285 not found: ID does not exist" containerID="e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285"
Sep 18 22:29:09.789513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.789454    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285"} err="failed to get container status \"e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285\": rpc error: code = NotFound desc = could not find container \"e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285\": container with ID starting with e5141800b90402c6d19a9d919a7691436467ea8c50c41eed62924548d7e7d285 not found: ID does not exist"
Sep 18 22:29:09.849893 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.849864    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:09.849893 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.849885    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp72h\" (UniqueName: \"kubernetes.io/projected/c6a26de2-4c45-4ec2-94a7-566d3f810667-kube-api-access-tp72h\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:09.849893 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.849896    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:09.850109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.849904    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:09.850109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.849913    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a26de2-4c45-4ec2-94a7-566d3f810667-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:09.850109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.849920    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a26de2-4c45-4ec2-94a7-566d3f810667-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:09.850109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:09.849928    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a26de2-4c45-4ec2-94a7-566d3f810667-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:29:10.084576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:10.084545    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"]
Sep 18 22:29:10.089281 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:10.089254    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-56446fbdbd-4hjj6"]
Sep 18 22:29:10.974862 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:10.974828    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" path="/var/lib/kubelet/pods/c6a26de2-4c45-4ec2-94a7-566d3f810667/volumes"
Sep 18 22:29:47.790690 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.790654    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"]
Sep 18 22:29:47.791183 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.791051    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="storage-initializer"
Sep 18 22:29:47.791183 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.791065    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="storage-initializer"
Sep 18 22:29:47.791183 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.791079    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main"
Sep 18 22:29:47.791183 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.791084    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main"
Sep 18 22:29:47.791183 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.791143    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a26de2-4c45-4ec2-94a7-566d3f810667" containerName="main"
Sep 18 22:29:47.794364 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.794344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.797575 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.797554    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 18 22:29:47.797770 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.797557    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:29:47.797970 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.797938    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 22:29:47.798059 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.797988    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 22:29:47.798582 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.798563    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-dm7mz\""
Sep 18 22:29:47.798670 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.798610    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 22:29:47.826709 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.826680    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"]
Sep 18 22:29:47.873535 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d26r6\" (UniqueName: \"kubernetes.io/projected/d82ecc95-b397-4496-9db9-32f7ac362127-kube-api-access-d26r6\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.873665 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-dshm\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.873665 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d82ecc95-b397-4496-9db9-32f7ac362127-cabundle-cert\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.873665 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-home\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.873665 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82ecc95-b397-4496-9db9-32f7ac362127-tls-certs\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.873826 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-tmp-dir\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.873826 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-model-cache\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.873826 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.873755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-kserve-provision-location\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.974523 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-home\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.974523 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82ecc95-b397-4496-9db9-32f7ac362127-tls-certs\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.974755 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-tmp-dir\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.974755 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-model-cache\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.974755 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-kserve-provision-location\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.974913 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d26r6\" (UniqueName: \"kubernetes.io/projected/d82ecc95-b397-4496-9db9-32f7ac362127-kube-api-access-d26r6\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.974913 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-dshm\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.975048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974907    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-home\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.975048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d82ecc95-b397-4496-9db9-32f7ac362127-cabundle-cert\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.975048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.974994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-tmp-dir\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.975048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.975016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-model-cache\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.975268 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.975044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-kserve-provision-location\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.975595 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.975573    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d82ecc95-b397-4496-9db9-32f7ac362127-cabundle-cert\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.977003 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.976979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-dshm\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.977091 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.977008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82ecc95-b397-4496-9db9-32f7ac362127-tls-certs\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:47.983319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:47.983296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d26r6\" (UniqueName: \"kubernetes.io/projected/d82ecc95-b397-4496-9db9-32f7ac362127-kube-api-access-d26r6\") pod \"auth-disabled-test-kserve-667dbb797f-swx48\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:48.104719 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:48.104634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:48.388639 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:48.388615    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"]
Sep 18 22:29:48.390487 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:29:48.390455    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82ecc95_b397_4496_9db9_32f7ac362127.slice/crio-d47182ac7e733949d0c7771529bebe96f45fd69578b89b9f1997aee2a40aa5ae WatchSource:0}: Error finding container d47182ac7e733949d0c7771529bebe96f45fd69578b89b9f1997aee2a40aa5ae: Status 404 returned error can't find the container with id d47182ac7e733949d0c7771529bebe96f45fd69578b89b9f1997aee2a40aa5ae
Sep 18 22:29:48.975196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:48.975155    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" event={"ID":"d82ecc95-b397-4496-9db9-32f7ac362127","Type":"ContainerStarted","Data":"f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b"}
Sep 18 22:29:48.975696 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:48.975211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" event={"ID":"d82ecc95-b397-4496-9db9-32f7ac362127","Type":"ContainerStarted","Data":"d47182ac7e733949d0c7771529bebe96f45fd69578b89b9f1997aee2a40aa5ae"}
Sep 18 22:29:50.988687 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:50.988599    2582 generic.go:356] "Generic (PLEG): container finished" podID="d82ecc95-b397-4496-9db9-32f7ac362127" containerID="f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b" exitCode=0
Sep 18 22:29:50.989163 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:50.988676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" event={"ID":"d82ecc95-b397-4496-9db9-32f7ac362127","Type":"ContainerDied","Data":"f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b"}
Sep 18 22:29:51.997309 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:51.997273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" event={"ID":"d82ecc95-b397-4496-9db9-32f7ac362127","Type":"ContainerStarted","Data":"cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071"}
Sep 18 22:29:52.038357 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:52.038292    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podStartSLOduration=2.863584795 podStartE2EDuration="5.038275864s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.174691069s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:52.036597503 +0000 UTC m=+2349.069776387" watchObservedRunningTime="2026-09-18 22:29:52.038275864 +0000 UTC m=+2349.071454757"
Sep 18 22:29:52.105465 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:52.105421    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:58.105551 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:58.105510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:29:58.107226 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:29:58.107194    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:30:08.105487 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:08.105442    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:30:18.105793 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:18.105743    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:30:28.106142 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:28.106098    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:30:38.105964 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:38.105904    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:30:45.138968 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:45.138930    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:30:45.145175 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:45.145154    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:30:45.146692 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:45.146668    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:30:45.152912 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:45.152890    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:30:48.105855 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:48.105816    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:30:58.105444 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:30:58.105354    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:31:08.105295 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:31:08.105246    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:31:18.105404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:31:18.105351    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:31:28.105854 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:31:28.105809    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:31:38.105536 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:31:38.105487    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:31:48.115609 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:31:48.115573    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:31:48.123602 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:31:48.123577    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:31:52.876100 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:31:52.876068    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"]
Sep 18 22:32:07.898873 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:07.898803    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" containerID="cri-o://cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071" gracePeriod=45
Sep 18 22:32:08.116582 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.116530    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 18 22:32:08.645530 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.645504    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:32:08.720287 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720206    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82ecc95-b397-4496-9db9-32f7ac362127-tls-certs\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720287 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720278    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-dshm\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720331    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d82ecc95-b397-4496-9db9-32f7ac362127-cabundle-cert\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-tmp-dir\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720404    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-kserve-provision-location\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720430    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d26r6\" (UniqueName: \"kubernetes.io/projected/d82ecc95-b397-4496-9db9-32f7ac362127-kube-api-access-d26r6\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-home\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720796 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720560    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-model-cache\") pod \"d82ecc95-b397-4496-9db9-32f7ac362127\" (UID: \"d82ecc95-b397-4496-9db9-32f7ac362127\") "
Sep 18 22:32:08.720922 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.720883    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82ecc95-b397-4496-9db9-32f7ac362127-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:32:08.721076 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.721051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-model-cache" (OuterVolumeSpecName: "model-cache") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:08.721410 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.721378    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-home" (OuterVolumeSpecName: "home") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:08.722647 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.722617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82ecc95-b397-4496-9db9-32f7ac362127-kube-api-access-d26r6" (OuterVolumeSpecName: "kube-api-access-d26r6") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "kube-api-access-d26r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:32:08.722737 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.722708    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-dshm" (OuterVolumeSpecName: "dshm") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:08.723201 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.723183    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82ecc95-b397-4496-9db9-32f7ac362127-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:32:08.733755 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.733730    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:08.745208 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.745182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d82ecc95-b397-4496-9db9-32f7ac362127" (UID: "d82ecc95-b397-4496-9db9-32f7ac362127"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:08.757326 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.757294    2582 generic.go:356] "Generic (PLEG): container finished" podID="d82ecc95-b397-4496-9db9-32f7ac362127" containerID="cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071" exitCode=0
Sep 18 22:32:08.757412 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.757347    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"
Sep 18 22:32:08.757412 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.757366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" event={"ID":"d82ecc95-b397-4496-9db9-32f7ac362127","Type":"ContainerDied","Data":"cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071"}
Sep 18 22:32:08.757489 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.757413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48" event={"ID":"d82ecc95-b397-4496-9db9-32f7ac362127","Type":"ContainerDied","Data":"d47182ac7e733949d0c7771529bebe96f45fd69578b89b9f1997aee2a40aa5ae"}
Sep 18 22:32:08.757489 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.757436    2582 scope.go:117] "RemoveContainer" containerID="cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071"
Sep 18 22:32:08.767487 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.767468    2582 scope.go:117] "RemoveContainer" containerID="f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b"
Sep 18 22:32:08.775409 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.775391    2582 scope.go:117] "RemoveContainer" containerID="cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071"
Sep 18 22:32:08.775645 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:32:08.775626    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071\": container with ID starting with cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071 not found: ID does not exist" containerID="cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071"
Sep 18 22:32:08.775713 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.775658    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071"} err="failed to get container status \"cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071\": rpc error: code = NotFound desc = could not find container \"cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071\": container with ID starting with cc280df6a2748264ad16a4a09575501fb559a10eef91d5847808a59688e65071 not found: ID does not exist"
Sep 18 22:32:08.775713 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.775682    2582 scope.go:117] "RemoveContainer" containerID="f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b"
Sep 18 22:32:08.775995 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:32:08.775978    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b\": container with ID starting with f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b not found: ID does not exist" containerID="f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b"
Sep 18 22:32:08.776040 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.776001    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b"} err="failed to get container status \"f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b\": rpc error: code = NotFound desc = could not find container \"f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b\": container with ID starting with f44eb8ceb8eec048c8597d16e9e49851f08366ab590724990b290a1ce81a751b not found: ID does not exist"
Sep 18 22:32:08.786534 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.786511    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"]
Sep 18 22:32:08.788644 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.788626    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-667dbb797f-swx48"]
Sep 18 22:32:08.821960 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.821926    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82ecc95-b397-4496-9db9-32f7ac362127-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.821960 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.821960    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.822068 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.821969    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d82ecc95-b397-4496-9db9-32f7ac362127-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.822068 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.821977    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.822068 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.821985    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.822068 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.821995    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d26r6\" (UniqueName: \"kubernetes.io/projected/d82ecc95-b397-4496-9db9-32f7ac362127-kube-api-access-d26r6\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.822068 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.822004    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.822068 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.822014    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82ecc95-b397-4496-9db9-32f7ac362127-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:32:08.975186 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:08.975102    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" path="/var/lib/kubelet/pods/d82ecc95-b397-4496-9db9-32f7ac362127/volumes"
Sep 18 22:32:43.223271 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.223241    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"]
Sep 18 22:32:43.223799 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.223780    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main"
Sep 18 22:32:43.223846 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.223801    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main"
Sep 18 22:32:43.223846 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.223818    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="storage-initializer"
Sep 18 22:32:43.223846 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.223824    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="storage-initializer"
Sep 18 22:32:43.223963 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.223889    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d82ecc95-b397-4496-9db9-32f7ac362127" containerName="main"
Sep 18 22:32:43.226931 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.226913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.231559 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.231539    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 18 22:32:43.231703 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.231539    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 18 22:32:43.231703 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.231592    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-fzbk6\""
Sep 18 22:32:43.232410 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.232389    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 18 22:32:43.241256 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.241227    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"]
Sep 18 22:32:43.321427 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-certs\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321592 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-envoy\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321592 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-istio-token\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321592 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb2396-95f9-4474-9074-d916c729e279-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321592 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321556    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-data\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321731 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb2396-95f9-4474-9074-d916c729e279-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321731 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-credential-socket\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321731 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-socket\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.321731 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.321676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjtl\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-kube-api-access-mqjtl\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423095 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb2396-95f9-4474-9074-d916c729e279-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423095 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-credential-socket\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-socket\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjtl\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-kube-api-access-mqjtl\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-certs\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-envoy\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-istio-token\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb2396-95f9-4474-9074-d916c729e279-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-data\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423663 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-socket\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423663 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-credential-socket\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423663 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-certs\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.423828 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.423689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-data\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.424224 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.424202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb2396-95f9-4474-9074-d916c729e279-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.425543 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.425517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-envoy\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.425697 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.425682    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb2396-95f9-4474-9074-d916c729e279-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.431669 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.431643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjtl\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-kube-api-access-mqjtl\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.432068 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.432050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-istio-token\") pod \"router-gateway-1-openshift-default-5bd85fff9f-sxv56\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.538205 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.538106    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:43.818341 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.818315    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"]
Sep 18 22:32:43.820013 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:32:43.819977    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fb2396_95f9_4474_9074_d916c729e279.slice/crio-49b88ae00589e276be36b982bcfff5229f3c205296c77cd5a13114f24afd7e5c WatchSource:0}: Error finding container 49b88ae00589e276be36b982bcfff5229f3c205296c77cd5a13114f24afd7e5c: Status 404 returned error can't find the container with id 49b88ae00589e276be36b982bcfff5229f3c205296c77cd5a13114f24afd7e5c
Sep 18 22:32:43.822083 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.822052    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:43.822155 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.822117    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:43.822155 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.822146    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:43.950664 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.950625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" event={"ID":"d1fb2396-95f9-4474-9074-d916c729e279","Type":"ContainerStarted","Data":"7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9"}
Sep 18 22:32:43.950870 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.950682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" event={"ID":"d1fb2396-95f9-4474-9074-d916c729e279","Type":"ContainerStarted","Data":"49b88ae00589e276be36b982bcfff5229f3c205296c77cd5a13114f24afd7e5c"}
Sep 18 22:32:43.974144 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:43.974093    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" podStartSLOduration=0.974077094 podStartE2EDuration="974.077094ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:32:43.971858582 +0000 UTC m=+2521.005037477" watchObservedRunningTime="2026-09-18 22:32:43.974077094 +0000 UTC m=+2521.007255988"
Sep 18 22:32:44.538821 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:44.538783    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:44.544022 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:44.543990    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:44.962789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:44.959792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:32:44.962789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:32:44.960858    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:33:09.084158 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:09.084122    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1fb2396-95f9-4474-9074-d916c729e279" containerID="7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9" exitCode=0
Sep 18 22:33:09.084575 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:09.084195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" event={"ID":"d1fb2396-95f9-4474-9074-d916c729e279","Type":"ContainerDied","Data":"7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9"}
Sep 18 22:33:09.084619 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:09.084572    2582 scope.go:117] "RemoveContainer" containerID="7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9"
Sep 18 22:33:09.085496 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:09.085466    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:33:09.085566 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:09.085538    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:33:09.085606 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:09.085571    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:33:10.091449 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:10.091411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" event={"ID":"d1fb2396-95f9-4474-9074-d916c729e279","Type":"ContainerStarted","Data":"bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a"}
Sep 18 22:33:10.538750 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:10.538709    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:33:10.539904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:10.539869    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 22:33:11.538970 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:11.538894    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 22:33:12.539491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:12.539448    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 22:33:13.538600 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:13.538551    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 22:33:14.539195 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:14.539147    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 22:33:14.959871 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:14.959825    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:33:15.542679 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:15.542647    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:33:15.543715 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:33:15.543693    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:34:22.752642 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:22.752605    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"]
Sep 18 22:34:22.753174 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:22.752813    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy" containerID="cri-o://bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a" gracePeriod=30
Sep 18 22:34:28.007043 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.006988    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:34:28.146863 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.146830    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-socket\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.146874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqjtl\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-kube-api-access-mqjtl\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.146908    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb2396-95f9-4474-9074-d916c729e279-istio-podinfo\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.146935    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-data\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147016    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-istio-token\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147041    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-envoy\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147066    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-certs\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147408 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb2396-95f9-4474-9074-d916c729e279-istiod-ca-cert\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147408 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147124    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-credential-socket\") pod \"d1fb2396-95f9-4474-9074-d916c729e279\" (UID: \"d1fb2396-95f9-4474-9074-d916c729e279\") "
Sep 18 22:34:28.147408 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147236    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:28.147576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:28.147576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147485    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-socket\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.147685 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:28.147778 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147747    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fb2396-95f9-4474-9074-d916c729e279-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:34:28.147817 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.147584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:28.149426 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.149400    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1fb2396-95f9-4474-9074-d916c729e279-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:34:28.149541 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.149478    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-kube-api-access-mqjtl" (OuterVolumeSpecName: "kube-api-access-mqjtl") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "kube-api-access-mqjtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:28.149589 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.149551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:28.149650 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.149633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1fb2396-95f9-4474-9074-d916c729e279" (UID: "d1fb2396-95f9-4474-9074-d916c729e279"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:28.249012 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.248975    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-istio-token\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.249012 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.249007    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-envoy\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.249012 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.249019    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-workload-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.249251 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.249031    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb2396-95f9-4474-9074-d916c729e279-istiod-ca-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.249251 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.249043    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-credential-socket\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.249251 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.249059    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjtl\" (UniqueName: \"kubernetes.io/projected/d1fb2396-95f9-4474-9074-d916c729e279-kube-api-access-mqjtl\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.249251 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.249083    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb2396-95f9-4474-9074-d916c729e279-istio-podinfo\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.249251 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.249095    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb2396-95f9-4474-9074-d916c729e279-istio-data\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:34:28.524371 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.524308    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1fb2396-95f9-4474-9074-d916c729e279" containerID="bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a" exitCode=0
Sep 18 22:34:28.524576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.524376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" event={"ID":"d1fb2396-95f9-4474-9074-d916c729e279","Type":"ContainerDied","Data":"bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a"}
Sep 18 22:34:28.524576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.524479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56" event={"ID":"d1fb2396-95f9-4474-9074-d916c729e279","Type":"ContainerDied","Data":"49b88ae00589e276be36b982bcfff5229f3c205296c77cd5a13114f24afd7e5c"}
Sep 18 22:34:28.524576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.524526    2582 scope.go:117] "RemoveContainer" containerID="bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a"
Sep 18 22:34:28.524576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.524542    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"
Sep 18 22:34:28.541149 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.541128    2582 scope.go:117] "RemoveContainer" containerID="7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9"
Sep 18 22:34:28.549999 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.549976    2582 scope.go:117] "RemoveContainer" containerID="bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a"
Sep 18 22:34:28.550298 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:34:28.550276    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a\": container with ID starting with bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a not found: ID does not exist" containerID="bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a"
Sep 18 22:34:28.550402 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.550310    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a"} err="failed to get container status \"bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a\": rpc error: code = NotFound desc = could not find container \"bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a\": container with ID starting with bba47788152b40ea10284aa4c115e08a100b1d6116816989f650f3e5afcf1f5a not found: ID does not exist"
Sep 18 22:34:28.550402 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.550331    2582 scope.go:117] "RemoveContainer" containerID="7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9"
Sep 18 22:34:28.550650 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:34:28.550622    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9\": container with ID starting with 7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9 not found: ID does not exist" containerID="7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9"
Sep 18 22:34:28.550721 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.550659    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9"} err="failed to get container status \"7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9\": rpc error: code = NotFound desc = could not find container \"7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9\": container with ID starting with 7c5f47260a03865c73b954fd1ca4c780abcea56eb98b29d120d42d33394c0af9 not found: ID does not exist"
Sep 18 22:34:28.551253 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.551232    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"]
Sep 18 22:34:28.557646 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.557627    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bd85fff9f-sxv56"]
Sep 18 22:34:28.975043 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:28.975011    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1fb2396-95f9-4474-9074-d916c729e279" path="/var/lib/kubelet/pods/d1fb2396-95f9-4474-9074-d916c729e279/volumes"
Sep 18 22:34:30.142876 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.142847    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"]
Sep 18 22:34:30.143253 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.143226    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy"
Sep 18 22:34:30.143253 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.143238    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy"
Sep 18 22:34:30.143332 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.143299    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy"
Sep 18 22:34:30.143380 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.143370    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy"
Sep 18 22:34:30.143380 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.143379    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy"
Sep 18 22:34:30.143444 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.143426    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fb2396-95f9-4474-9074-d916c729e279" containerName="istio-proxy"
Sep 18 22:34:30.147764 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.147744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.150759 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.150734    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 18 22:34:30.150890 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.150757    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 18 22:34:30.150890 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.150739    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-dpvjp\""
Sep 18 22:34:30.150890 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.150766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 18 22:34:30.150890 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.150739    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:34:30.151122 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.150990    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 18 22:34:30.160343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.160321    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"]
Sep 18 22:34:30.266848 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.266810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv47n\" (UniqueName: \"kubernetes.io/projected/57cc036b-5205-4fa3-845d-03af4185f67e-kube-api-access-wv47n\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.266848 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.266853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.267116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.266875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc036b-5205-4fa3-845d-03af4185f67e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.267116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.266927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.267116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.267020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57cc036b-5205-4fa3-845d-03af4185f67e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.267116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.267047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.267116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.267075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.267116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.267093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv47n\" (UniqueName: \"kubernetes.io/projected/57cc036b-5205-4fa3-845d-03af4185f67e-kube-api-access-wv47n\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc036b-5205-4fa3-845d-03af4185f67e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368863 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57cc036b-5205-4fa3-845d-03af4185f67e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368863 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368671    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.368863 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.368725    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.369079 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.369056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.369159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.369137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.369336 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.369313    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57cc036b-5205-4fa3-845d-03af4185f67e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.371192 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.371163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.371454 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.371434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc036b-5205-4fa3-845d-03af4185f67e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.380828 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.380795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv47n\" (UniqueName: \"kubernetes.io/projected/57cc036b-5205-4fa3-845d-03af4185f67e-kube-api-access-wv47n\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.459554 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.459476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:30.746551 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:30.746461    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"]
Sep 18 22:34:30.750051 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:34:30.750020    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cc036b_5205_4fa3_845d_03af4185f67e.slice/crio-03b548ae74ccb5a507d17c188154ecda01d6ea8adce6f06bcd389a6a29310176 WatchSource:0}: Error finding container 03b548ae74ccb5a507d17c188154ecda01d6ea8adce6f06bcd389a6a29310176: Status 404 returned error can't find the container with id 03b548ae74ccb5a507d17c188154ecda01d6ea8adce6f06bcd389a6a29310176
Sep 18 22:34:31.547019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:31.546984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" event={"ID":"57cc036b-5205-4fa3-845d-03af4185f67e","Type":"ContainerStarted","Data":"bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886"}
Sep 18 22:34:31.547404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:31.547040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" event={"ID":"57cc036b-5205-4fa3-845d-03af4185f67e","Type":"ContainerStarted","Data":"03b548ae74ccb5a507d17c188154ecda01d6ea8adce6f06bcd389a6a29310176"}
Sep 18 22:34:33.562343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:33.562307    2582 generic.go:356] "Generic (PLEG): container finished" podID="57cc036b-5205-4fa3-845d-03af4185f67e" containerID="bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886" exitCode=0
Sep 18 22:34:33.562742 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:33.562359    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" event={"ID":"57cc036b-5205-4fa3-845d-03af4185f67e","Type":"ContainerDied","Data":"bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886"}
Sep 18 22:34:34.568986 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:34.568936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" event={"ID":"57cc036b-5205-4fa3-845d-03af4185f67e","Type":"ContainerStarted","Data":"750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5"}
Sep 18 22:34:34.594384 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:34.594330    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podStartSLOduration=2.391476233 podStartE2EDuration="4.594314649s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.202838416s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:34:34.591934356 +0000 UTC m=+2631.625113242" watchObservedRunningTime="2026-09-18 22:34:34.594314649 +0000 UTC m=+2631.627493543"
Sep 18 22:34:35.459605 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:35.459576    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:40.459691 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:40.459652    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:34:40.461403 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:40.461365    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:34:50.460703 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:34:50.460643    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:35:00.460423 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:00.460374    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:35:10.460768 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:10.460725    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:35:20.460296 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:20.460251    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:35:30.460743 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:30.460650    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:35:40.460276 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:40.460230    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:35:45.182793 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:45.182763    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:35:45.189100 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:45.189077    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:35:45.191126 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:45.191109    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:35:45.196921 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:45.196902    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:35:50.460432 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:35:50.460395    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:36:00.460797 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:00.460751    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:36:10.460751 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:10.460706    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:36:20.460249 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:20.460200    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:36:30.469743 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:30.469713    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:36:30.477474 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:30.477445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:36:36.022870 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:36.022836    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"]
Sep 18 22:36:51.046796 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.046739    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" containerID="cri-o://750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5" gracePeriod=45
Sep 18 22:36:51.471027 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.470986    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 18 22:36:51.794859 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.794828    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:36:51.950203 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv47n\" (UniqueName: \"kubernetes.io/projected/57cc036b-5205-4fa3-845d-03af4185f67e-kube-api-access-wv47n\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950231    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-model-cache\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950252    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-dshm\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950284    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-kserve-provision-location\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950319    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57cc036b-5205-4fa3-845d-03af4185f67e-cabundle-cert\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950338    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-home\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950361    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc036b-5205-4fa3-845d-03af4185f67e-tls-certs\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-tmp-dir\") pod \"57cc036b-5205-4fa3-845d-03af4185f67e\" (UID: \"57cc036b-5205-4fa3-845d-03af4185f67e\") "
Sep 18 22:36:51.950895 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950764    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cc036b-5205-4fa3-845d-03af4185f67e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:36:51.951005 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.950985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-model-cache" (OuterVolumeSpecName: "model-cache") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:51.951228 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.951202    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-home" (OuterVolumeSpecName: "home") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:51.953015 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.952972    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cc036b-5205-4fa3-845d-03af4185f67e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:36:51.953181 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.953149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-dshm" (OuterVolumeSpecName: "dshm") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:51.953248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.953165    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cc036b-5205-4fa3-845d-03af4185f67e-kube-api-access-wv47n" (OuterVolumeSpecName: "kube-api-access-wv47n") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "kube-api-access-wv47n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:36:51.969510 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.969481    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:51.979579 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:51.979556    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57cc036b-5205-4fa3-845d-03af4185f67e" (UID: "57cc036b-5205-4fa3-845d-03af4185f67e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:52.051677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051601    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv47n\" (UniqueName: \"kubernetes.io/projected/57cc036b-5205-4fa3-845d-03af4185f67e-kube-api-access-wv47n\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.051677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051628    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.051677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051642    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.051677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051654    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.051677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051667    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57cc036b-5205-4fa3-845d-03af4185f67e-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.051677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051679    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.052201 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051691    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cc036b-5205-4fa3-845d-03af4185f67e-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.052201 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.051723    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cc036b-5205-4fa3-845d-03af4185f67e-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:36:52.356244 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.356152    2582 generic.go:356] "Generic (PLEG): container finished" podID="57cc036b-5205-4fa3-845d-03af4185f67e" containerID="750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5" exitCode=0
Sep 18 22:36:52.356244 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.356201    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"
Sep 18 22:36:52.356244 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.356201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" event={"ID":"57cc036b-5205-4fa3-845d-03af4185f67e","Type":"ContainerDied","Data":"750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5"}
Sep 18 22:36:52.356514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.356272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr" event={"ID":"57cc036b-5205-4fa3-845d-03af4185f67e","Type":"ContainerDied","Data":"03b548ae74ccb5a507d17c188154ecda01d6ea8adce6f06bcd389a6a29310176"}
Sep 18 22:36:52.356514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.356304    2582 scope.go:117] "RemoveContainer" containerID="750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5"
Sep 18 22:36:52.366010 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.365988    2582 scope.go:117] "RemoveContainer" containerID="bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886"
Sep 18 22:36:52.373753 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.373735    2582 scope.go:117] "RemoveContainer" containerID="750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5"
Sep 18 22:36:52.374034 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:36:52.374015    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5\": container with ID starting with 750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5 not found: ID does not exist" containerID="750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5"
Sep 18 22:36:52.374118 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.374047    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5"} err="failed to get container status \"750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5\": rpc error: code = NotFound desc = could not find container \"750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5\": container with ID starting with 750b1d3db36bf726a75f180e9b08625d9e3d117629f793e99593cfaf9aa428c5 not found: ID does not exist"
Sep 18 22:36:52.374118 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.374072    2582 scope.go:117] "RemoveContainer" containerID="bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886"
Sep 18 22:36:52.374339 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:36:52.374321    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886\": container with ID starting with bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886 not found: ID does not exist" containerID="bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886"
Sep 18 22:36:52.374382 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.374346    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886"} err="failed to get container status \"bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886\": rpc error: code = NotFound desc = could not find container \"bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886\": container with ID starting with bacbe0881c9d4cf657bf074ccbfc94c52202ea2ce7f7c2c8fcde980782cc8886 not found: ID does not exist"
Sep 18 22:36:52.379377 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.379355    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"]
Sep 18 22:36:52.383455 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.383430    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-785b89bdd86ntlr"]
Sep 18 22:36:52.974111 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:36:52.974075    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" path="/var/lib/kubelet/pods/57cc036b-5205-4fa3-845d-03af4185f67e/volumes"
Sep 18 22:38:31.594290 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:31.590107    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7_0341eae0-7d5e-4ccd-a01c-846a7dccfb2e/llm-d-routing-sidecar/0.log"
Sep 18 22:38:31.598590 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:31.598565    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-40b85f95_scheduler-v06-threshold-migration-test-kserve-59b84dfb55-qpnd7_0341eae0-7d5e-4ccd-a01c-846a7dccfb2e/main/0.log"
Sep 18 22:38:40.434237 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.434203    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"]
Sep 18 22:38:40.434693 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.434562    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="storage-initializer"
Sep 18 22:38:40.434693 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.434573    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="storage-initializer"
Sep 18 22:38:40.434693 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.434584    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main"
Sep 18 22:38:40.434693 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.434589    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main"
Sep 18 22:38:40.434693 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.434656    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cc036b-5205-4fa3-845d-03af4185f67e" containerName="main"
Sep 18 22:38:40.437680 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.437659    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.440746 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.440580    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:38:40.440887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.440842    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-b9thv\""
Sep 18 22:38:40.441383 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.441343    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 22:38:40.441506 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.441490    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 22:38:40.441556 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.441526    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 22:38:40.441714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.441701    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 22:38:40.453027 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.453002    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"]
Sep 18 22:38:40.566310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-home\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.566310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.566537 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a11d19e-2788-478a-9884-b93f3221454d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.566537 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.566537 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhtc\" (UniqueName: \"kubernetes.io/projected/6a11d19e-2788-478a-9884-b93f3221454d-kube-api-access-tqhtc\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.566537 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.566537 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a11d19e-2788-478a-9884-b93f3221454d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.566705 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.566552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-dshm\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667272 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a11d19e-2788-478a-9884-b93f3221454d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667272 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-dshm\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-home\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a11d19e-2788-478a-9884-b93f3221454d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhtc\" (UniqueName: \"kubernetes.io/projected/6a11d19e-2788-478a-9884-b93f3221454d-kube-api-access-tqhtc\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667846 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667806    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-home\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.667924 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.668016 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.667915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.668264 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.668240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a11d19e-2788-478a-9884-b93f3221454d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.668343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.668249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.669706 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.669683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-dshm\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.669891 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.669875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a11d19e-2788-478a-9884-b93f3221454d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.676440 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.676417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhtc\" (UniqueName: \"kubernetes.io/projected/6a11d19e-2788-478a-9884-b93f3221454d-kube-api-access-tqhtc\") pod \"tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.724395 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.724263    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"]
Sep 18 22:38:40.730094 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.730070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.747431 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.747402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"]
Sep 18 22:38:40.748973 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.748893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:40.868607 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.868567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.868607 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.868610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.868856 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.868651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.868856 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.868753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.969202 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.969163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.969399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.969262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.969399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.969318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.969399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.969342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.969737 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.969710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.969826 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.969756    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.970007 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.969985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:40.972212 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:40.972192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:41.031159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:41.031131    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"]
Sep 18 22:38:41.031733 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:38:41.031705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a11d19e_2788_478a_9884_b93f3221454d.slice/crio-0fbf4d2085192548f953829b63102b907932cfc1f9068f972f673b894e99463b WatchSource:0}: Error finding container 0fbf4d2085192548f953829b63102b907932cfc1f9068f972f673b894e99463b: Status 404 returned error can't find the container with id 0fbf4d2085192548f953829b63102b907932cfc1f9068f972f673b894e99463b
Sep 18 22:38:41.042937 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:41.042917    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:38:41.320094 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:41.320069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"]
Sep 18 22:38:41.321901 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:38:41.321870    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fb639b_be54_4b7e_be67_6fbdf37ae4c3.slice/crio-f41ff80e6a3605335e8b7eb64197458ccd96c0d960eed487dc7007f0b071f85a WatchSource:0}: Error finding container f41ff80e6a3605335e8b7eb64197458ccd96c0d960eed487dc7007f0b071f85a: Status 404 returned error can't find the container with id f41ff80e6a3605335e8b7eb64197458ccd96c0d960eed487dc7007f0b071f85a
Sep 18 22:38:41.937321 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:41.937281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" event={"ID":"13fb639b-be54-4b7e-be67-6fbdf37ae4c3","Type":"ContainerStarted","Data":"54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4"}
Sep 18 22:38:41.937785 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:41.937383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" event={"ID":"13fb639b-be54-4b7e-be67-6fbdf37ae4c3","Type":"ContainerStarted","Data":"f41ff80e6a3605335e8b7eb64197458ccd96c0d960eed487dc7007f0b071f85a"}
Sep 18 22:38:41.939276 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:41.939244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" event={"ID":"6a11d19e-2788-478a-9884-b93f3221454d","Type":"ContainerStarted","Data":"7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa"}
Sep 18 22:38:41.939388 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:41.939301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" event={"ID":"6a11d19e-2788-478a-9884-b93f3221454d","Type":"ContainerStarted","Data":"0fbf4d2085192548f953829b63102b907932cfc1f9068f972f673b894e99463b"}
Sep 18 22:38:43.953558 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:43.953529    2582 generic.go:356] "Generic (PLEG): container finished" podID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerID="54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4" exitCode=0
Sep 18 22:38:43.954005 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:43.953602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" event={"ID":"13fb639b-be54-4b7e-be67-6fbdf37ae4c3","Type":"ContainerDied","Data":"54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4"}
Sep 18 22:38:43.955017 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:43.954998    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:38:43.955294 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:43.955274    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a11d19e-2788-478a-9884-b93f3221454d" containerID="7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa" exitCode=0
Sep 18 22:38:43.955392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:43.955365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" event={"ID":"6a11d19e-2788-478a-9884-b93f3221454d","Type":"ContainerDied","Data":"7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa"}
Sep 18 22:38:44.965338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:44.965293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" event={"ID":"6a11d19e-2788-478a-9884-b93f3221454d","Type":"ContainerStarted","Data":"01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8"}
Sep 18 22:38:44.993650 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:44.993583    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" podStartSLOduration=2.501845524 podStartE2EDuration="4.993560833s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.491715309s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:38:44.991408166 +0000 UTC m=+2882.024587060" watchObservedRunningTime="2026-09-18 22:38:44.993560833 +0000 UTC m=+2882.026739729"
Sep 18 22:38:45.749620 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:45.749585    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:50.749835 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:50.749732    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:50.768242 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:50.768210    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:38:51.032087 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:38:51.032006    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:39:31.287509 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:39:31.287472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" event={"ID":"13fb639b-be54-4b7e-be67-6fbdf37ae4c3","Type":"ContainerStarted","Data":"424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8"}
Sep 18 22:39:31.288094 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:39:31.287680    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:40:02.304022 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:02.303917    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:40:02.344937 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:02.344880    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" podStartSLOduration=33.320714461 podStartE2EDuration="1m22.344863341s" totalImagesPullingTime="46.484503764s" totalInitContainerRuntime="2.539645116s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:39:31.331548907 +0000 UTC m=+2928.364727801" watchObservedRunningTime="2026-09-18 22:40:02.344863341 +0000 UTC m=+2959.378042236"
Sep 18 22:40:03.561630 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:03.561604    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2_13fb639b-be54-4b7e-be67-6fbdf37ae4c3/main/0.log"
Sep 18 22:40:04.589174 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:04.589138    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"]
Sep 18 22:40:04.598424 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:40:04.598393    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:40:04.598424 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:40:04.598424 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:40:04.598424 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8" cmd=["/bin/sleep","15"]
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:04.598440    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" message="" err=<
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:40:04.598452    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="main" containerID="cri-o://01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8"
Sep 18 22:40:04.598685 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:04.598486    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="main" containerID="cri-o://01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8" gracePeriod=60
Sep 18 22:40:04.608258 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:04.608232    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"]
Sep 18 22:40:04.608466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:04.608443    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" podUID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerName="main" containerID="cri-o://424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8" gracePeriod=30
Sep 18 22:40:05.016682 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.016648    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 22:40:05.184306 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.184279    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:40:05.254042 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.253938    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tmp-dir\") pod \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") "
Sep 18 22:40:05.254042 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.254007    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-cabundle-cert\") pod \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") "
Sep 18 22:40:05.254042 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.254034    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-kserve-provision-location\") pod \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") "
Sep 18 22:40:05.254274 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.254136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tls-certs\") pod \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\" (UID: \"13fb639b-be54-4b7e-be67-6fbdf37ae4c3\") "
Sep 18 22:40:05.254392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.254363    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13fb639b-be54-4b7e-be67-6fbdf37ae4c3" (UID: "13fb639b-be54-4b7e-be67-6fbdf37ae4c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:40:05.254805 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.254783    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13fb639b-be54-4b7e-be67-6fbdf37ae4c3" (UID: "13fb639b-be54-4b7e-be67-6fbdf37ae4c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.256765 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.256738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13fb639b-be54-4b7e-be67-6fbdf37ae4c3" (UID: "13fb639b-be54-4b7e-be67-6fbdf37ae4c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:40:05.279019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.278985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13fb639b-be54-4b7e-be67-6fbdf37ae4c3" (UID: "13fb639b-be54-4b7e-be67-6fbdf37ae4c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.343552 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.343525    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:40:05.355095 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.355070    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.355095 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.355094    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.355266 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.355103    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.355266 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.355112    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13fb639b-be54-4b7e-be67-6fbdf37ae4c3-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.461686 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.461561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqhtc\" (UniqueName: \"kubernetes.io/projected/6a11d19e-2788-478a-9884-b93f3221454d-kube-api-access-tqhtc\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.461686 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.461663    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-model-cache\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.461970 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.461745    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-tmp-dir\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.461970 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.461782    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-kserve-provision-location\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.461970 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.461824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-dshm\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.461970 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.461881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-home\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.462212 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462020    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.462212 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.462212 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a11d19e-2788-478a-9884-b93f3221454d-cabundle-cert\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.462212 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462169    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a11d19e-2788-478a-9884-b93f3221454d-tls-certs\") pod \"6a11d19e-2788-478a-9884-b93f3221454d\" (UID: \"6a11d19e-2788-478a-9884-b93f3221454d\") "
Sep 18 22:40:05.462212 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462171    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-home" (OuterVolumeSpecName: "home") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.462657 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462630    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a11d19e-2788-478a-9884-b93f3221454d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:40:05.462939 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462914    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.463077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462961    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a11d19e-2788-478a-9884-b93f3221454d-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.463077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462979    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.463077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.462991    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.464025 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.463997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a11d19e-2788-478a-9884-b93f3221454d-kube-api-access-tqhtc" (OuterVolumeSpecName: "kube-api-access-tqhtc") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "kube-api-access-tqhtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:40:05.464662 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.464644    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-dshm" (OuterVolumeSpecName: "dshm") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.464926 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.464914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a11d19e-2788-478a-9884-b93f3221454d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:40:05.485691 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.485663    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a11d19e-2788-478a-9884-b93f3221454d" (UID: "6a11d19e-2788-478a-9884-b93f3221454d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:40:05.489595 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.489569    2582 generic.go:356] "Generic (PLEG): container finished" podID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerID="424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8" exitCode=0
Sep 18 22:40:05.489705 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.489630    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"
Sep 18 22:40:05.489705 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.489649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" event={"ID":"13fb639b-be54-4b7e-be67-6fbdf37ae4c3","Type":"ContainerDied","Data":"424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8"}
Sep 18 22:40:05.489815 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.489745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2" event={"ID":"13fb639b-be54-4b7e-be67-6fbdf37ae4c3","Type":"ContainerDied","Data":"f41ff80e6a3605335e8b7eb64197458ccd96c0d960eed487dc7007f0b071f85a"}
Sep 18 22:40:05.489815 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.489776    2582 scope.go:117] "RemoveContainer" containerID="424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8"
Sep 18 22:40:05.491412 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.491388    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a11d19e-2788-478a-9884-b93f3221454d" containerID="01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8" exitCode=0
Sep 18 22:40:05.491516 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.491453    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"
Sep 18 22:40:05.491516 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.491474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" event={"ID":"6a11d19e-2788-478a-9884-b93f3221454d","Type":"ContainerDied","Data":"01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8"}
Sep 18 22:40:05.491613 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.491528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg" event={"ID":"6a11d19e-2788-478a-9884-b93f3221454d","Type":"ContainerDied","Data":"0fbf4d2085192548f953829b63102b907932cfc1f9068f972f673b894e99463b"}
Sep 18 22:40:05.500714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.500693    2582 scope.go:117] "RemoveContainer" containerID="54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4"
Sep 18 22:40:05.511809 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.511791    2582 scope.go:117] "RemoveContainer" containerID="424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8"
Sep 18 22:40:05.512123 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:40:05.512103    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8\": container with ID starting with 424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8 not found: ID does not exist" containerID="424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8"
Sep 18 22:40:05.512186 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.512131    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8"} err="failed to get container status \"424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8\": rpc error: code = NotFound desc = could not find container \"424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8\": container with ID starting with 424310912fbfb8580f8e263d0adfc5687ea486e5622ec0b63c47b6cdefae0fb8 not found: ID does not exist"
Sep 18 22:40:05.512186 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.512151    2582 scope.go:117] "RemoveContainer" containerID="54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4"
Sep 18 22:40:05.512448 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:40:05.512424    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4\": container with ID starting with 54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4 not found: ID does not exist" containerID="54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4"
Sep 18 22:40:05.512492 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.512457    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4"} err="failed to get container status \"54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4\": rpc error: code = NotFound desc = could not find container \"54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4\": container with ID starting with 54f359aaddb774d294c57da86a3964956b380013d27f26d80fa3653874d031b4 not found: ID does not exist"
Sep 18 22:40:05.512492 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.512483    2582 scope.go:117] "RemoveContainer" containerID="01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8"
Sep 18 22:40:05.520996 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.520980    2582 scope.go:117] "RemoveContainer" containerID="7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa"
Sep 18 22:40:05.528919 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.528901    2582 scope.go:117] "RemoveContainer" containerID="01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8"
Sep 18 22:40:05.529253 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:40:05.529234    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8\": container with ID starting with 01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8 not found: ID does not exist" containerID="01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8"
Sep 18 22:40:05.529329 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.529258    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8"} err="failed to get container status \"01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8\": rpc error: code = NotFound desc = could not find container \"01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8\": container with ID starting with 01c21ad7299d4325403be5b169b0bc7eeb68c5e5a86eacbf73844277d0cf9db8 not found: ID does not exist"
Sep 18 22:40:05.529329 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.529275    2582 scope.go:117] "RemoveContainer" containerID="7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa"
Sep 18 22:40:05.529528 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:40:05.529504    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa\": container with ID starting with 7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa not found: ID does not exist" containerID="7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa"
Sep 18 22:40:05.529616 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.529530    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa"} err="failed to get container status \"7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa\": rpc error: code = NotFound desc = could not find container \"7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa\": container with ID starting with 7e83f9abb92cd2b3d363da52cd0d6ed3128f56e109538a235dcbc982b801a6aa not found: ID does not exist"
Sep 18 22:40:05.532792 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.532769    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"]
Sep 18 22:40:05.540691 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.540670    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5d6f5fdbf7-cbppg"]
Sep 18 22:40:05.563413 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.563389    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhtc\" (UniqueName: \"kubernetes.io/projected/6a11d19e-2788-478a-9884-b93f3221454d-kube-api-access-tqhtc\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.563413 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.563410    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.563541 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.563421    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a11d19e-2788-478a-9884-b93f3221454d-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.563541 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.563430    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a11d19e-2788-478a-9884-b93f3221454d-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:40:05.569654 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.569634    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"]
Sep 18 22:40:05.577200 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:05.577178    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-688dfc5b8d-p72d2"]
Sep 18 22:40:06.974264 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:06.974228    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" path="/var/lib/kubelet/pods/13fb639b-be54-4b7e-be67-6fbdf37ae4c3/volumes"
Sep 18 22:40:06.974720 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:06.974705    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a11d19e-2788-478a-9884-b93f3221454d" path="/var/lib/kubelet/pods/6a11d19e-2788-478a-9884-b93f3221454d/volumes"
Sep 18 22:40:45.220776 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:45.220751    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:40:45.226365 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:45.226347    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:40:45.229495 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:45.229472    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:40:45.235082 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:40:45.235066    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:41:00.192735 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.192704    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"]
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193123    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="main"
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193138    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="main"
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193150    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerName="main"
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193156    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerName="main"
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193172    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerName="storage-initializer"
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193178    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerName="storage-initializer"
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193187    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="storage-initializer"
Sep 18 22:41:00.193196 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193193    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="storage-initializer"
Sep 18 22:41:00.193478 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193253    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="13fb639b-be54-4b7e-be67-6fbdf37ae4c3" containerName="main"
Sep 18 22:41:00.193478 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.193260    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a11d19e-2788-478a-9884-b93f3221454d" containerName="main"
Sep 18 22:41:00.197866 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.197845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.201430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.201408    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tnktd\""
Sep 18 22:41:00.201550 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.201441    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 22:41:00.201550 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.201454    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 22:41:00.201550 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.201459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 22:41:00.201550 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.201495    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:41:00.201772 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.201682    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 22:41:00.216241 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.216219    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"]
Sep 18 22:41:00.326782 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.326746    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.326782 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.326786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-model-cache\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.327003 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.326812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.327003 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.326874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-dshm\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.327003 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.326992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvgl\" (UniqueName: \"kubernetes.io/projected/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kube-api-access-6kvgl\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.327109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.327034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.327109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.327054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.327109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.327087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-home\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428133 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvgl\" (UniqueName: \"kubernetes.io/projected/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kube-api-access-6kvgl\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-home\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-model-cache\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428310 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-dshm\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428691 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428691 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428813 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-model-cache\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428813 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-home\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.428934 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.428918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.430586 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.430567    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-dshm\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.430964 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.430932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.440754 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.440727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvgl\" (UniqueName: \"kubernetes.io/projected/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kube-api-access-6kvgl\") pod \"tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.458323 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.458264    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"]
Sep 18 22:41:00.462924 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.462904    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.483129 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.483106    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"]
Sep 18 22:41:00.508967 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.508921    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:00.529058 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.529031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.529174 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.529064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39e38c5-8eb5-4251-8e57-218f52a4d6db-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.529174 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.529089    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.529276 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.529214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.630244 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.630196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.630406 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.630318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.630406 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.630345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39e38c5-8eb5-4251-8e57-218f52a4d6db-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.630406 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.630374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.630691 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.630673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.630730 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.630688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.631055 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.631037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39e38c5-8eb5-4251-8e57-218f52a4d6db-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.632803 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.632781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.773805 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.773704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:00.822333 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:00.822178    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"]
Sep 18 22:41:00.825838 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:41:00.825711    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbac122_e5fa_4ca2_b0fb_f0911bc91687.slice/crio-6cc198573d5659e1be719787a12d5f2f5dc2c85e1655eac385aa6e96cf21b2fb WatchSource:0}: Error finding container 6cc198573d5659e1be719787a12d5f2f5dc2c85e1655eac385aa6e96cf21b2fb: Status 404 returned error can't find the container with id 6cc198573d5659e1be719787a12d5f2f5dc2c85e1655eac385aa6e96cf21b2fb
Sep 18 22:41:01.074269 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:01.074237    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"]
Sep 18 22:41:01.074442 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:41:01.074416    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39e38c5_8eb5_4251_8e57_218f52a4d6db.slice/crio-93474ed9bfbab444e065cec0848fa206201d23f8ffd3acd3c4186c0bbb8fdae2 WatchSource:0}: Error finding container 93474ed9bfbab444e065cec0848fa206201d23f8ffd3acd3c4186c0bbb8fdae2: Status 404 returned error can't find the container with id 93474ed9bfbab444e065cec0848fa206201d23f8ffd3acd3c4186c0bbb8fdae2
Sep 18 22:41:01.788161 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:01.788119    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" event={"ID":"e39e38c5-8eb5-4251-8e57-218f52a4d6db","Type":"ContainerStarted","Data":"578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54"}
Sep 18 22:41:01.788657 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:01.788180    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" event={"ID":"e39e38c5-8eb5-4251-8e57-218f52a4d6db","Type":"ContainerStarted","Data":"93474ed9bfbab444e065cec0848fa206201d23f8ffd3acd3c4186c0bbb8fdae2"}
Sep 18 22:41:01.789838 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:01.789807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" event={"ID":"bcbac122-e5fa-4ca2-b0fb-f0911bc91687","Type":"ContainerStarted","Data":"578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb"}
Sep 18 22:41:01.789984 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:01.789851    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" event={"ID":"bcbac122-e5fa-4ca2-b0fb-f0911bc91687","Type":"ContainerStarted","Data":"6cc198573d5659e1be719787a12d5f2f5dc2c85e1655eac385aa6e96cf21b2fb"}
Sep 18 22:41:03.804478 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:03.804445    2582 generic.go:356] "Generic (PLEG): container finished" podID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerID="578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54" exitCode=0
Sep 18 22:41:03.804971 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:03.804528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" event={"ID":"e39e38c5-8eb5-4251-8e57-218f52a4d6db","Type":"ContainerDied","Data":"578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54"}
Sep 18 22:41:03.806236 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:03.806217    2582 generic.go:356] "Generic (PLEG): container finished" podID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerID="578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb" exitCode=0
Sep 18 22:41:03.806343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:03.806256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" event={"ID":"bcbac122-e5fa-4ca2-b0fb-f0911bc91687","Type":"ContainerDied","Data":"578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb"}
Sep 18 22:41:04.816112 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:04.816069    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" event={"ID":"e39e38c5-8eb5-4251-8e57-218f52a4d6db","Type":"ContainerStarted","Data":"32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce"}
Sep 18 22:41:04.816590 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:04.816320    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:04.818239 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:04.818212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" event={"ID":"bcbac122-e5fa-4ca2-b0fb-f0911bc91687","Type":"ContainerStarted","Data":"9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c"}
Sep 18 22:41:04.842491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:04.842419    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" podStartSLOduration=2.302529457 podStartE2EDuration="4.842402408s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.539872951s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:41:04.839792346 +0000 UTC m=+3021.872971242" watchObservedRunningTime="2026-09-18 22:41:04.842402408 +0000 UTC m=+3021.875581301"
Sep 18 22:41:04.866872 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:04.866821    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" podStartSLOduration=2.2839230710000002 podStartE2EDuration="4.866801549s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.582878478s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:41:04.864001268 +0000 UTC m=+3021.897180162" watchObservedRunningTime="2026-09-18 22:41:04.866801549 +0000 UTC m=+3021.899980443"
Sep 18 22:41:05.509721 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:05.509679    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:10.509737 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:10.509691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:10.522742 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:10.522718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:10.864442 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:10.864417    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:41:35.836468 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:35.836374    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:41:39.229773 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:41:39.229742    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt_e39e38c5-8eb5-4251-8e57-218f52a4d6db/main/0.log"
Sep 18 22:42:01.421680 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.421629    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"]
Sep 18 22:42:01.430859 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.430830    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"]
Sep 18 22:42:01.431138 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.431092    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" podUID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerName="main" containerID="cri-o://32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce" gracePeriod=30
Sep 18 22:42:01.432155 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:42:01.432125    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:42:01.432155 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:42:01.432155 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:42:01.432155 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c" cmd=["/bin/sleep","15"]
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.432184    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" message="" err=<
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:42:01.432201    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerName="main" containerID="cri-o://9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c"
Sep 18 22:42:01.432450 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.432242    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerName="main" containerID="cri-o://9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c" gracePeriod=60
Sep 18 22:42:01.788790 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.788762    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:42:01.903074 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903037    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kserve-provision-location\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903081    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-home\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903148    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-cabundle-cert\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903180    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-dshm\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903208    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kvgl\" (UniqueName: \"kubernetes.io/projected/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kube-api-access-6kvgl\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903237    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tls-certs\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903505 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903285    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tmp-dir\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903505 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903332    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-model-cache\") pod \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\" (UID: \"bcbac122-e5fa-4ca2-b0fb-f0911bc91687\") "
Sep 18 22:42:01.903505 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903352    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-home" (OuterVolumeSpecName: "home") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.903661 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.903661 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903597    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:01.903661 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:42:01.903824 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.903807    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.905451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.905431    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-dshm" (OuterVolumeSpecName: "dshm") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:01.905616 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.905594    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kube-api-access-6kvgl" (OuterVolumeSpecName: "kube-api-access-6kvgl") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "kube-api-access-6kvgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:01.905774 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.905751    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:01.925961 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:01.925916    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bcbac122-e5fa-4ca2-b0fb-f0911bc91687" (UID: "bcbac122-e5fa-4ca2-b0fb-f0911bc91687"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:02.004262 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.004231    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.004389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.004266    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.004389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.004283    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.004389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.004296    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvgl\" (UniqueName: \"kubernetes.io/projected/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-kube-api-access-6kvgl\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.004389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.004310    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.004389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.004345    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.004389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.004358    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcbac122-e5fa-4ca2-b0fb-f0911bc91687-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.072880 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.072852    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:42:02.122236 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.122202    2582 generic.go:356] "Generic (PLEG): container finished" podID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerID="32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce" exitCode=0
Sep 18 22:42:02.122405 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.122257    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"
Sep 18 22:42:02.122405 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.122286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" event={"ID":"e39e38c5-8eb5-4251-8e57-218f52a4d6db","Type":"ContainerDied","Data":"32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce"}
Sep 18 22:42:02.122405 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.122345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt" event={"ID":"e39e38c5-8eb5-4251-8e57-218f52a4d6db","Type":"ContainerDied","Data":"93474ed9bfbab444e065cec0848fa206201d23f8ffd3acd3c4186c0bbb8fdae2"}
Sep 18 22:42:02.122405 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.122377    2582 scope.go:117] "RemoveContainer" containerID="32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce"
Sep 18 22:42:02.124255 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.124225    2582 generic.go:356] "Generic (PLEG): container finished" podID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerID="9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c" exitCode=0
Sep 18 22:42:02.124381 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.124278    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"
Sep 18 22:42:02.124381 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.124279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" event={"ID":"bcbac122-e5fa-4ca2-b0fb-f0911bc91687","Type":"ContainerDied","Data":"9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c"}
Sep 18 22:42:02.124381 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.124331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg" event={"ID":"bcbac122-e5fa-4ca2-b0fb-f0911bc91687","Type":"ContainerDied","Data":"6cc198573d5659e1be719787a12d5f2f5dc2c85e1655eac385aa6e96cf21b2fb"}
Sep 18 22:42:02.132351 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.132334    2582 scope.go:117] "RemoveContainer" containerID="578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54"
Sep 18 22:42:02.141006 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.140991    2582 scope.go:117] "RemoveContainer" containerID="32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce"
Sep 18 22:42:02.141284 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:42:02.141266    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce\": container with ID starting with 32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce not found: ID does not exist" containerID="32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce"
Sep 18 22:42:02.141343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.141291    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce"} err="failed to get container status \"32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce\": rpc error: code = NotFound desc = could not find container \"32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce\": container with ID starting with 32659631eaf15532d3a69399335275233e94c3a7be60c1e20139f82a520f85ce not found: ID does not exist"
Sep 18 22:42:02.141343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.141306    2582 scope.go:117] "RemoveContainer" containerID="578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54"
Sep 18 22:42:02.141538 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:42:02.141522    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54\": container with ID starting with 578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54 not found: ID does not exist" containerID="578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54"
Sep 18 22:42:02.141583 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.141539    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54"} err="failed to get container status \"578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54\": rpc error: code = NotFound desc = could not find container \"578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54\": container with ID starting with 578448e1be675f2748dbaf27a460e8936642962f78097133556af8a5ec58ed54 not found: ID does not exist"
Sep 18 22:42:02.141583 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.141550    2582 scope.go:117] "RemoveContainer" containerID="9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c"
Sep 18 22:42:02.145685 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.145659    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"]
Sep 18 22:42:02.150276 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.150252    2582 scope.go:117] "RemoveContainer" containerID="578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb"
Sep 18 22:42:02.152222 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.152205    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7dd75fb95b-5s2zg"]
Sep 18 22:42:02.158371 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.158354    2582 scope.go:117] "RemoveContainer" containerID="9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c"
Sep 18 22:42:02.158626 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:42:02.158608    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c\": container with ID starting with 9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c not found: ID does not exist" containerID="9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c"
Sep 18 22:42:02.158677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.158632    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c"} err="failed to get container status \"9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c\": rpc error: code = NotFound desc = could not find container \"9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c\": container with ID starting with 9dcc6b8f08cd585253edcb752b76cfd06f495248cb51cc0dba303d82e4529c9c not found: ID does not exist"
Sep 18 22:42:02.158677 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.158649    2582 scope.go:117] "RemoveContainer" containerID="578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb"
Sep 18 22:42:02.158898 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:42:02.158876    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb\": container with ID starting with 578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb not found: ID does not exist" containerID="578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb"
Sep 18 22:42:02.158985 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.158909    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb"} err="failed to get container status \"578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb\": rpc error: code = NotFound desc = could not find container \"578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb\": container with ID starting with 578fd08e66056832dfb4d80c55d74ac4fde6ece1c1bc2dbdc342feebd72b9cfb not found: ID does not exist"
Sep 18 22:42:02.206442 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.206414    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tmp-dir\") pod \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") "
Sep 18 22:42:02.206599 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.206452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39e38c5-8eb5-4251-8e57-218f52a4d6db-cabundle-cert\") pod \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") "
Sep 18 22:42:02.206599 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.206492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-kserve-provision-location\") pod \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") "
Sep 18 22:42:02.206599 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.206529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tls-certs\") pod \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\" (UID: \"e39e38c5-8eb5-4251-8e57-218f52a4d6db\") "
Sep 18 22:42:02.206859 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.206831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e39e38c5-8eb5-4251-8e57-218f52a4d6db" (UID: "e39e38c5-8eb5-4251-8e57-218f52a4d6db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:02.206911 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.206840    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39e38c5-8eb5-4251-8e57-218f52a4d6db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e39e38c5-8eb5-4251-8e57-218f52a4d6db" (UID: "e39e38c5-8eb5-4251-8e57-218f52a4d6db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:42:02.208608 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.208588    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e39e38c5-8eb5-4251-8e57-218f52a4d6db" (UID: "e39e38c5-8eb5-4251-8e57-218f52a4d6db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:02.229333 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.229307    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e39e38c5-8eb5-4251-8e57-218f52a4d6db" (UID: "e39e38c5-8eb5-4251-8e57-218f52a4d6db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:02.307378 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.307285    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.307378 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.307315    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e39e38c5-8eb5-4251-8e57-218f52a4d6db-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.307378 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.307325    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e39e38c5-8eb5-4251-8e57-218f52a4d6db-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.307378 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.307335    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39e38c5-8eb5-4251-8e57-218f52a4d6db-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:42:02.445580 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.445542    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"]
Sep 18 22:42:02.449819 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.449792    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-9847d5958-hbbgt"]
Sep 18 22:42:02.974538 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.974502    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" path="/var/lib/kubelet/pods/bcbac122-e5fa-4ca2-b0fb-f0911bc91687/volumes"
Sep 18 22:42:02.974964 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:02.974937    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" path="/var/lib/kubelet/pods/e39e38c5-8eb5-4251-8e57-218f52a4d6db/volumes"
Sep 18 22:42:58.123694 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.123618    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"]
Sep 18 22:42:58.124256 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124212    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerName="storage-initializer"
Sep 18 22:42:58.124256 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124233    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerName="storage-initializer"
Sep 18 22:42:58.124256 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124250    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerName="main"
Sep 18 22:42:58.124429 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124259    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerName="main"
Sep 18 22:42:58.124429 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124277    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerName="main"
Sep 18 22:42:58.124429 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124285    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerName="main"
Sep 18 22:42:58.124429 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124295    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerName="storage-initializer"
Sep 18 22:42:58.124429 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124304    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerName="storage-initializer"
Sep 18 22:42:58.124429 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124422    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcbac122-e5fa-4ca2-b0fb-f0911bc91687" containerName="main"
Sep 18 22:42:58.124720 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.124435    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39e38c5-8eb5-4251-8e57-218f52a4d6db" containerName="main"
Sep 18 22:42:58.128991 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.128970    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.131375 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.131355    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 22:42:58.131375 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.131365    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 22:42:58.131543 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.131525    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 22:42:58.131609 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.131596    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7wfdb\""
Sep 18 22:42:58.132378 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.132362    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:42:58.132462 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.132396    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 22:42:58.139856 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.139835    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"]
Sep 18 22:42:58.258618 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.258794 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-dshm\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.258794 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21639a29-40d0-41de-ac1c-98674fcb058a-tls-certs\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.258794 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-model-cache\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.258794 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-home\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.259048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21639a29-40d0-41de-ac1c-98674fcb058a-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.259048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-tmp-dir\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.259048 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.258911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdbc\" (UniqueName: \"kubernetes.io/projected/21639a29-40d0-41de-ac1c-98674fcb058a-kube-api-access-pqdbc\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.347709 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.347647    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"]
Sep 18 22:42:58.351513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.351490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.359475 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-model-cache\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.359626 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-home\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.359626 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21639a29-40d0-41de-ac1c-98674fcb058a-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.359626 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-tmp-dir\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.359626 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdbc\" (UniqueName: \"kubernetes.io/projected/21639a29-40d0-41de-ac1c-98674fcb058a-kube-api-access-pqdbc\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.359626 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.360434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359904    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-home\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.360434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.359984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-model-cache\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.360434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.360002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.360434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.360032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-dshm\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.360434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.360073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21639a29-40d0-41de-ac1c-98674fcb058a-tls-certs\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.360434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.360212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-tmp-dir\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.360434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.360389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21639a29-40d0-41de-ac1c-98674fcb058a-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.363319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.363288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21639a29-40d0-41de-ac1c-98674fcb058a-tls-certs\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.363436 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.363413    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"]
Sep 18 22:42:58.364635 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.364613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-dshm\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.374517 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.374445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdbc\" (UniqueName: \"kubernetes.io/projected/21639a29-40d0-41de-ac1c-98674fcb058a-kube-api-access-pqdbc\") pod \"tokenizer-template-args-test-kserve-748cd6c88d-lq6fk\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.440042 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.440014    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:42:58.460896 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.460861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.461111 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.460914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f786e18e-38f0-4de3-843b-d2cff81d0ea2-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.461111 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.460934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.461111 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.461026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.561709 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.561681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.561869 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.561766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.561869 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.561795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f786e18e-38f0-4de3-843b-d2cff81d0ea2-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.561869 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.561820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.562110 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.562092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.562144 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.562119    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.562508 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.562487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f786e18e-38f0-4de3-843b-d2cff81d0ea2-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.564399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.564380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.663149 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.663052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:42:58.724213 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.724187    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"]
Sep 18 22:42:58.725774 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:42:58.725748    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21639a29_40d0_41de_ac1c_98674fcb058a.slice/crio-4a854c8ce1a852d27234e057a494373efe18fc469ecd3e1092dd89d87b6b8663 WatchSource:0}: Error finding container 4a854c8ce1a852d27234e057a494373efe18fc469ecd3e1092dd89d87b6b8663: Status 404 returned error can't find the container with id 4a854c8ce1a852d27234e057a494373efe18fc469ecd3e1092dd89d87b6b8663
Sep 18 22:42:58.938491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:58.938461    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"]
Sep 18 22:42:58.940193 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:42:58.940165    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf786e18e_38f0_4de3_843b_d2cff81d0ea2.slice/crio-7372b0393b1f27bbe238ed4966835e4d9ba30feeb75bb41a662c547f7ee598b3 WatchSource:0}: Error finding container 7372b0393b1f27bbe238ed4966835e4d9ba30feeb75bb41a662c547f7ee598b3: Status 404 returned error can't find the container with id 7372b0393b1f27bbe238ed4966835e4d9ba30feeb75bb41a662c547f7ee598b3
Sep 18 22:42:59.426986 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:59.426930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" event={"ID":"21639a29-40d0-41de-ac1c-98674fcb058a","Type":"ContainerStarted","Data":"01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6"}
Sep 18 22:42:59.427498 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:59.427013    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" event={"ID":"21639a29-40d0-41de-ac1c-98674fcb058a","Type":"ContainerStarted","Data":"4a854c8ce1a852d27234e057a494373efe18fc469ecd3e1092dd89d87b6b8663"}
Sep 18 22:42:59.428638 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:59.428607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" event={"ID":"f786e18e-38f0-4de3-843b-d2cff81d0ea2","Type":"ContainerStarted","Data":"01fb574b1dd0f3bc47b8dba91af940995e6d5955140f33f9b54c1be75758eb1d"}
Sep 18 22:42:59.428765 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:42:59.428658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" event={"ID":"f786e18e-38f0-4de3-843b-d2cff81d0ea2","Type":"ContainerStarted","Data":"7372b0393b1f27bbe238ed4966835e4d9ba30feeb75bb41a662c547f7ee598b3"}
Sep 18 22:43:01.440292 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:01.440258    2582 generic.go:356] "Generic (PLEG): container finished" podID="21639a29-40d0-41de-ac1c-98674fcb058a" containerID="01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6" exitCode=0
Sep 18 22:43:01.440654 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:01.440309    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" event={"ID":"21639a29-40d0-41de-ac1c-98674fcb058a","Type":"ContainerDied","Data":"01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6"}
Sep 18 22:43:02.447027 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:02.446992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" event={"ID":"21639a29-40d0-41de-ac1c-98674fcb058a","Type":"ContainerStarted","Data":"0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec"}
Sep 18 22:43:02.448537 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:02.448512    2582 generic.go:356] "Generic (PLEG): container finished" podID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerID="01fb574b1dd0f3bc47b8dba91af940995e6d5955140f33f9b54c1be75758eb1d" exitCode=0
Sep 18 22:43:02.448658 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:02.448561    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" event={"ID":"f786e18e-38f0-4de3-843b-d2cff81d0ea2","Type":"ContainerDied","Data":"01fb574b1dd0f3bc47b8dba91af940995e6d5955140f33f9b54c1be75758eb1d"}
Sep 18 22:43:02.480623 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:02.480581    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" podStartSLOduration=1.937127303 podStartE2EDuration="4.480543747s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.543416444s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:43:02.479695026 +0000 UTC m=+3139.512873919" watchObservedRunningTime="2026-09-18 22:43:02.480543747 +0000 UTC m=+3139.513722640"
Sep 18 22:43:03.440674 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:03.440636    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:43:03.455692 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:03.455656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" event={"ID":"f786e18e-38f0-4de3-843b-d2cff81d0ea2","Type":"ContainerStarted","Data":"254a157a14c6d899f64ffbbcd71b9530f11fdb7ca19a689b17461b1a6ecaf2b2"}
Sep 18 22:43:03.456109 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:03.456085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:43:03.475455 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:03.475252    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" podStartSLOduration=2.9554526819999998 podStartE2EDuration="5.475229307s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.519776625s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:43:03.472484936 +0000 UTC m=+3140.505663830" watchObservedRunningTime="2026-09-18 22:43:03.475229307 +0000 UTC m=+3140.508408202"
Sep 18 22:43:08.440808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:08.440768    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:43:08.453849 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:08.453818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:43:08.500372 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:08.500339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:43:34.474620 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:34.474575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:43:35.547905 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:35.547848    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv_f786e18e-38f0-4de3-843b-d2cff81d0ea2/main/0.log"
Sep 18 22:43:36.067319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.067112    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"]
Sep 18 22:43:36.070514 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.070486    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"]
Sep 18 22:43:36.070725 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.070696    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" podUID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerName="main" containerID="cri-o://254a157a14c6d899f64ffbbcd71b9530f11fdb7ca19a689b17461b1a6ecaf2b2" gracePeriod=30
Sep 18 22:43:36.078453 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:43:36.078417    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:43:36.078453 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:43:36.078453 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:43:36.078453 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec" cmd=["/bin/sleep","15"]
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.078469    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" message="" err=<
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:43:36.078481    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" containerName="main" containerID="cri-o://0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec"
Sep 18 22:43:36.078689 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.078512    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" containerName="main" containerID="cri-o://0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec" gracePeriod=60
Sep 18 22:43:36.426464 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.426439    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:43:36.509984 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.509933    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-dshm\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-home\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510040    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-kserve-provision-location\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21639a29-40d0-41de-ac1c-98674fcb058a-tls-certs\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510266 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510121    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqdbc\" (UniqueName: \"kubernetes.io/projected/21639a29-40d0-41de-ac1c-98674fcb058a-kube-api-access-pqdbc\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510266 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510167    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21639a29-40d0-41de-ac1c-98674fcb058a-cabundle-cert\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510266 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-tmp-dir\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510266 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510257    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-model-cache\") pod \"21639a29-40d0-41de-ac1c-98674fcb058a\" (UID: \"21639a29-40d0-41de-ac1c-98674fcb058a\") "
Sep 18 22:43:36.510741 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.510709    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-model-cache" (OuterVolumeSpecName: "model-cache") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.511051 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.511025    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-home" (OuterVolumeSpecName: "home") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.511289 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.511268    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21639a29-40d0-41de-ac1c-98674fcb058a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:36.511425 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.511407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.515363 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.515333    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21639a29-40d0-41de-ac1c-98674fcb058a-kube-api-access-pqdbc" (OuterVolumeSpecName: "kube-api-access-pqdbc") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "kube-api-access-pqdbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:36.515459 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.515363    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-dshm" (OuterVolumeSpecName: "dshm") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.515459 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.515384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21639a29-40d0-41de-ac1c-98674fcb058a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:43:36.532186 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.532165    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21639a29-40d0-41de-ac1c-98674fcb058a" (UID: "21639a29-40d0-41de-ac1c-98674fcb058a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.611398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611325    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.611398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611357    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.611398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611365    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.611398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611373    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.611398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611383    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21639a29-40d0-41de-ac1c-98674fcb058a-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.611398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611392    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21639a29-40d0-41de-ac1c-98674fcb058a-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.611398 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611400    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqdbc\" (UniqueName: \"kubernetes.io/projected/21639a29-40d0-41de-ac1c-98674fcb058a-kube-api-access-pqdbc\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.611903 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.611410    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21639a29-40d0-41de-ac1c-98674fcb058a-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.650019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.649982    2582 generic.go:356] "Generic (PLEG): container finished" podID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerID="254a157a14c6d899f64ffbbcd71b9530f11fdb7ca19a689b17461b1a6ecaf2b2" exitCode=0
Sep 18 22:43:36.650154 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.650038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" event={"ID":"f786e18e-38f0-4de3-843b-d2cff81d0ea2","Type":"ContainerDied","Data":"254a157a14c6d899f64ffbbcd71b9530f11fdb7ca19a689b17461b1a6ecaf2b2"}
Sep 18 22:43:36.652116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.652083    2582 generic.go:356] "Generic (PLEG): container finished" podID="21639a29-40d0-41de-ac1c-98674fcb058a" containerID="0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec" exitCode=0
Sep 18 22:43:36.652233 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.652128    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"
Sep 18 22:43:36.652233 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.652132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" event={"ID":"21639a29-40d0-41de-ac1c-98674fcb058a","Type":"ContainerDied","Data":"0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec"}
Sep 18 22:43:36.652233 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.652183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk" event={"ID":"21639a29-40d0-41de-ac1c-98674fcb058a","Type":"ContainerDied","Data":"4a854c8ce1a852d27234e057a494373efe18fc469ecd3e1092dd89d87b6b8663"}
Sep 18 22:43:36.652233 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.652222    2582 scope.go:117] "RemoveContainer" containerID="0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec"
Sep 18 22:43:36.664576 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.664556    2582 scope.go:117] "RemoveContainer" containerID="01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6"
Sep 18 22:43:36.674824 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.674799    2582 scope.go:117] "RemoveContainer" containerID="0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec"
Sep 18 22:43:36.675222 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:43:36.675201    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec\": container with ID starting with 0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec not found: ID does not exist" containerID="0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec"
Sep 18 22:43:36.675312 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.675232    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec"} err="failed to get container status \"0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec\": rpc error: code = NotFound desc = could not find container \"0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec\": container with ID starting with 0ead877455bd1f639eefa78b5b2f333dd692f4878c5fc34dabb3309aa13d87ec not found: ID does not exist"
Sep 18 22:43:36.675312 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.675249    2582 scope.go:117] "RemoveContainer" containerID="01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6"
Sep 18 22:43:36.675486 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:43:36.675473    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6\": container with ID starting with 01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6 not found: ID does not exist" containerID="01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6"
Sep 18 22:43:36.675535 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.675488    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6"} err="failed to get container status \"01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6\": rpc error: code = NotFound desc = could not find container \"01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6\": container with ID starting with 01a9784c1a0fa48a32fd39123a8dc36bd4ba3f041ccd2448157f3b5f15cf97a6 not found: ID does not exist"
Sep 18 22:43:36.680992 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.680968    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"]
Sep 18 22:43:36.684396 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.684373    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-748cd6c88d-lq6fk"]
Sep 18 22:43:36.712582 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.712562    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:43:36.813752 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.813715    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f786e18e-38f0-4de3-843b-d2cff81d0ea2-cabundle-cert\") pod \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") "
Sep 18 22:43:36.813922 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.813803    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tls-certs\") pod \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") "
Sep 18 22:43:36.813922 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.813833    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-kserve-provision-location\") pod \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") "
Sep 18 22:43:36.813922 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.813878    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tmp-dir\") pod \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\" (UID: \"f786e18e-38f0-4de3-843b-d2cff81d0ea2\") "
Sep 18 22:43:36.814224 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.814201    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f786e18e-38f0-4de3-843b-d2cff81d0ea2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f786e18e-38f0-4de3-843b-d2cff81d0ea2" (UID: "f786e18e-38f0-4de3-843b-d2cff81d0ea2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:36.814350 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.814326    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f786e18e-38f0-4de3-843b-d2cff81d0ea2" (UID: "f786e18e-38f0-4de3-843b-d2cff81d0ea2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.815981 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.815956    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f786e18e-38f0-4de3-843b-d2cff81d0ea2" (UID: "f786e18e-38f0-4de3-843b-d2cff81d0ea2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:43:36.836288 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.836259    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f786e18e-38f0-4de3-843b-d2cff81d0ea2" (UID: "f786e18e-38f0-4de3-843b-d2cff81d0ea2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.915381 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.915356    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.915381 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.915380    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.915540 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.915390    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f786e18e-38f0-4de3-843b-d2cff81d0ea2-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.915540 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.915401    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f786e18e-38f0-4de3-843b-d2cff81d0ea2-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.974263 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:36.974232    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" path="/var/lib/kubelet/pods/21639a29-40d0-41de-ac1c-98674fcb058a/volumes"
Sep 18 22:43:37.660336 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:37.660303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv" event={"ID":"f786e18e-38f0-4de3-843b-d2cff81d0ea2","Type":"ContainerDied","Data":"7372b0393b1f27bbe238ed4966835e4d9ba30feeb75bb41a662c547f7ee598b3"}
Sep 18 22:43:37.660753 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:37.660352    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"
Sep 18 22:43:37.660753 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:37.660362    2582 scope.go:117] "RemoveContainer" containerID="254a157a14c6d899f64ffbbcd71b9530f11fdb7ca19a689b17461b1a6ecaf2b2"
Sep 18 22:43:37.670234 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:37.670211    2582 scope.go:117] "RemoveContainer" containerID="01fb574b1dd0f3bc47b8dba91af940995e6d5955140f33f9b54c1be75758eb1d"
Sep 18 22:43:37.677459 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:37.677435    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"]
Sep 18 22:43:37.680611 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:37.680590    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7db4464b5d-rf7wv"]
Sep 18 22:43:38.974510 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:43:38.974466    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" path="/var/lib/kubelet/pods/f786e18e-38f0-4de3-843b-d2cff81d0ea2/volumes"
Sep 18 22:44:34.309724 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.309642    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"]
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310075    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerName="main"
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310089    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerName="main"
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310106    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" containerName="main"
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310112    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" containerName="main"
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310131    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" containerName="storage-initializer"
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310138    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" containerName="storage-initializer"
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310147    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerName="storage-initializer"
Sep 18 22:44:34.310178 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310152    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerName="storage-initializer"
Sep 18 22:44:34.310430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310228    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f786e18e-38f0-4de3-843b-d2cff81d0ea2" containerName="main"
Sep 18 22:44:34.310430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.310239    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="21639a29-40d0-41de-ac1c-98674fcb058a" containerName="main"
Sep 18 22:44:34.313362 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.313345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.317119 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.317093    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-v57kr\""
Sep 18 22:44:34.317119 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.317105    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-7tjwp\""
Sep 18 22:44:34.317327 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.317150    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 18 22:44:34.317327 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.317106    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 18 22:44:34.317327 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.317213    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 18 22:44:34.321596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.321574    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"]
Sep 18 22:44:34.438718 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.438673    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlth6\" (UniqueName: \"kubernetes.io/projected/6c4692c9-68eb-4e76-92d6-8043b4298de9-kube-api-access-qlth6\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.438896 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.438782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4692c9-68eb-4e76-92d6-8043b4298de9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.539591 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.539553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4692c9-68eb-4e76-92d6-8043b4298de9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.539754 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.539603    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlth6\" (UniqueName: \"kubernetes.io/projected/6c4692c9-68eb-4e76-92d6-8043b4298de9-kube-api-access-qlth6\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.542165 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.542141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4692c9-68eb-4e76-92d6-8043b4298de9-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.547838 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.547811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlth6\" (UniqueName: \"kubernetes.io/projected/6c4692c9-68eb-4e76-92d6-8043b4298de9-kube-api-access-qlth6\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.624572 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.624542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:34.898792 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.898762    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"]
Sep 18 22:44:34.900326 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:44:34.900299    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4692c9_68eb_4e76_92d6_8043b4298de9.slice/crio-d7070223e10820d91857750638e319f53c799c0c1cf6b8ca7c811af0bba0b6d3 WatchSource:0}: Error finding container d7070223e10820d91857750638e319f53c799c0c1cf6b8ca7c811af0bba0b6d3: Status 404 returned error can't find the container with id d7070223e10820d91857750638e319f53c799c0c1cf6b8ca7c811af0bba0b6d3
Sep 18 22:44:34.902418 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.902401    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:44:34.964531 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:34.964492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" event={"ID":"6c4692c9-68eb-4e76-92d6-8043b4298de9","Type":"ContainerStarted","Data":"d7070223e10820d91857750638e319f53c799c0c1cf6b8ca7c811af0bba0b6d3"}
Sep 18 22:44:36.977529 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:36.977490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" event={"ID":"6c4692c9-68eb-4e76-92d6-8043b4298de9","Type":"ContainerStarted","Data":"13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0"}
Sep 18 22:44:36.978032 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:36.977656    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:44:36.994667 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:44:36.994608    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" podStartSLOduration=1.622991649 podStartE2EDuration="2.994591457s" totalImagesPullingTime="1.371599808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:44:36.99382314 +0000 UTC m=+3234.027002036" watchObservedRunningTime="2026-09-18 22:44:36.994591457 +0000 UTC m=+3234.027770351"
Sep 18 22:45:07.985244 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:07.985214    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:45:09.468017 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:09.467977    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"]
Sep 18 22:45:17.985459 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:17.985407    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:45:24.488657 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:24.488614    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main" containerID="cri-o://13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0" gracePeriod=45
Sep 18 22:45:27.985017 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:27.984975    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:45:37.984699 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:37.984662    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:45:37.985105 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:37.984775    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:45:38.320177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.320096    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"]
Sep 18 22:45:38.324341 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.324321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.326972 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.326935    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 18 22:45:38.327103 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.326975    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 22:45:38.327103 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.326923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:45:38.327103 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.327012    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-pm4c9\""
Sep 18 22:45:38.327287 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.327100    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 18 22:45:38.327349 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.327318    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 22:45:38.334634 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.334611    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"]
Sep 18 22:45:38.494479 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-dshm\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.494682 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920dfec1-ebeb-42b4-9d69-1bf654bca46f-cabundle-cert\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.494682 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-model-cache\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.494682 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-home\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.494682 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494673    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tls-certs\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.494861 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb46c\" (UniqueName: \"kubernetes.io/projected/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kube-api-access-sb46c\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.494861 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tmp-dir\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.494861 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.494757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kserve-provision-location\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920dfec1-ebeb-42b4-9d69-1bf654bca46f-cabundle-cert\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595246 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-model-cache\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-home\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tls-certs\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb46c\" (UniqueName: \"kubernetes.io/projected/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kube-api-access-sb46c\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tmp-dir\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kserve-provision-location\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595752 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-dshm\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595752 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-model-cache\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595874 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tmp-dir\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595919 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-home\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.595919 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.595899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kserve-provision-location\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.596134 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.596112    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920dfec1-ebeb-42b4-9d69-1bf654bca46f-cabundle-cert\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.597765 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.597745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-dshm\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.597898 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.597880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tls-certs\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.603485 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.603463    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb46c\" (UniqueName: \"kubernetes.io/projected/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kube-api-access-sb46c\") pod \"router-with-refs-test-kserve-75f6996fb5-tpvrw\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.636332 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.636303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:38.920766 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:38.920732    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"]
Sep 18 22:45:38.921955 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:45:38.921913    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920dfec1_ebeb_42b4_9d69_1bf654bca46f.slice/crio-691535c638d6b9ba71975f26f3cb42e6528acb132d917d76020c19a212abe53a WatchSource:0}: Error finding container 691535c638d6b9ba71975f26f3cb42e6528acb132d917d76020c19a212abe53a: Status 404 returned error can't find the container with id 691535c638d6b9ba71975f26f3cb42e6528acb132d917d76020c19a212abe53a
Sep 18 22:45:39.310216 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:39.310127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" event={"ID":"920dfec1-ebeb-42b4-9d69-1bf654bca46f","Type":"ContainerStarted","Data":"36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07"}
Sep 18 22:45:39.310216 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:39.310195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" event={"ID":"920dfec1-ebeb-42b4-9d69-1bf654bca46f","Type":"ContainerStarted","Data":"691535c638d6b9ba71975f26f3cb42e6528acb132d917d76020c19a212abe53a"}
Sep 18 22:45:41.325128 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:41.325094    2582 generic.go:356] "Generic (PLEG): container finished" podID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerID="36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07" exitCode=0
Sep 18 22:45:41.325524 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:41.325176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" event={"ID":"920dfec1-ebeb-42b4-9d69-1bf654bca46f","Type":"ContainerDied","Data":"36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07"}
Sep 18 22:45:42.332609 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:42.332570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" event={"ID":"920dfec1-ebeb-42b4-9d69-1bf654bca46f","Type":"ContainerStarted","Data":"99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9"}
Sep 18 22:45:42.357874 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:42.357817    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podStartSLOduration=2.175599205 podStartE2EDuration="4.357802493s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.182203288s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:45:42.355098233 +0000 UTC m=+3299.388277128" watchObservedRunningTime="2026-09-18 22:45:42.357802493 +0000 UTC m=+3299.390981387"
Sep 18 22:45:42.636866 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:42.636824    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:45.261119 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:45.261092    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:45:45.267333 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:45.267296    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:45:45.272334 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:45.272316    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:45:45.278276 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:45.278259    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:45:47.985500 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:47.985457    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 22:45:48.637317 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:48.637272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:45:48.638897 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:48.638866    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:45:54.750326 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:54.750265    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:45:54.832113 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:54.832057    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4692c9-68eb-4e76-92d6-8043b4298de9-tls-certs\") pod \"6c4692c9-68eb-4e76-92d6-8043b4298de9\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") "
Sep 18 22:45:54.832113 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:54.832122    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlth6\" (UniqueName: \"kubernetes.io/projected/6c4692c9-68eb-4e76-92d6-8043b4298de9-kube-api-access-qlth6\") pod \"6c4692c9-68eb-4e76-92d6-8043b4298de9\" (UID: \"6c4692c9-68eb-4e76-92d6-8043b4298de9\") "
Sep 18 22:45:54.834317 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:54.834283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4692c9-68eb-4e76-92d6-8043b4298de9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c4692c9-68eb-4e76-92d6-8043b4298de9" (UID: "6c4692c9-68eb-4e76-92d6-8043b4298de9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:45:54.834451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:54.834341    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4692c9-68eb-4e76-92d6-8043b4298de9-kube-api-access-qlth6" (OuterVolumeSpecName: "kube-api-access-qlth6") pod "6c4692c9-68eb-4e76-92d6-8043b4298de9" (UID: "6c4692c9-68eb-4e76-92d6-8043b4298de9"). InnerVolumeSpecName "kube-api-access-qlth6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:45:54.933922 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:54.933880    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlth6\" (UniqueName: \"kubernetes.io/projected/6c4692c9-68eb-4e76-92d6-8043b4298de9-kube-api-access-qlth6\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:45:54.933922 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:54.933921    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c4692c9-68eb-4e76-92d6-8043b4298de9-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:45:55.416781 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.416743    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerID="13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0" exitCode=0
Sep 18 22:45:55.416989 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.416809    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"
Sep 18 22:45:55.416989 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.416836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" event={"ID":"6c4692c9-68eb-4e76-92d6-8043b4298de9","Type":"ContainerDied","Data":"13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0"}
Sep 18 22:45:55.416989 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.416891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq" event={"ID":"6c4692c9-68eb-4e76-92d6-8043b4298de9","Type":"ContainerDied","Data":"d7070223e10820d91857750638e319f53c799c0c1cf6b8ca7c811af0bba0b6d3"}
Sep 18 22:45:55.416989 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.416914    2582 scope.go:117] "RemoveContainer" containerID="13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0"
Sep 18 22:45:55.426670 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.426651    2582 scope.go:117] "RemoveContainer" containerID="13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0"
Sep 18 22:45:55.426974 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:45:55.426936    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0\": container with ID starting with 13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0 not found: ID does not exist" containerID="13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0"
Sep 18 22:45:55.427058 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.426983    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0"} err="failed to get container status \"13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0\": rpc error: code = NotFound desc = could not find container \"13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0\": container with ID starting with 13ec2bf58a3e5690c7985ad8a9d7a1873dfef8db12ba49f65c1ec401d224f8c0 not found: ID does not exist"
Sep 18 22:45:55.435066 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.435045    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"]
Sep 18 22:45:55.438869 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:55.438846    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schet6djq"]
Sep 18 22:45:56.975431 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:56.975396    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" path="/var/lib/kubelet/pods/6c4692c9-68eb-4e76-92d6-8043b4298de9/volumes"
Sep 18 22:45:58.637385 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:45:58.637340    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:46:08.637341 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:46:08.637278    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:46:18.637542 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:46:18.637494    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:46:28.636699 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:46:28.636647    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:46:38.637053 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:46:38.637000    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:46:48.637535 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:46:48.637489    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:46:58.637472 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:46:58.637421    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:47:08.637638 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:47:08.637593    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:47:18.636765 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:47:18.636714    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:47:28.637192 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:47:28.637092    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:47:38.647211 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:47:38.647177    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:47:38.655061 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:47:38.655028    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:47:48.148105 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:47:48.148071    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"]
Sep 18 22:48:03.171070 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.171021    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" containerID="cri-o://99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9" gracePeriod=45
Sep 18 22:48:03.648588 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.648546    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 22:48:03.913307 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.913236    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:48:03.994209 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994173    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-dshm\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994209 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994216    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tmp-dir\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994482 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994246    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-model-cache\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994482 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994306    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-home\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994482 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994340    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb46c\" (UniqueName: \"kubernetes.io/projected/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kube-api-access-sb46c\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994482 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kserve-provision-location\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994482 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tls-certs\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994482 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994450    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920dfec1-ebeb-42b4-9d69-1bf654bca46f-cabundle-cert\") pod \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\" (UID: \"920dfec1-ebeb-42b4-9d69-1bf654bca46f\") "
Sep 18 22:48:03.994787 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.994558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-model-cache" (OuterVolumeSpecName: "model-cache") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:03.996619 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.996382    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:03.996619 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.996440    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920dfec1-ebeb-42b4-9d69-1bf654bca46f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:48:03.996619 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:03.996556    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-home" (OuterVolumeSpecName: "home") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:04.003859 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.003813    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:04.004212 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.004169    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-dshm" (OuterVolumeSpecName: "dshm") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:04.005140 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.005096    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:48:04.005402 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.005367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kube-api-access-sb46c" (OuterVolumeSpecName: "kube-api-access-sb46c") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "kube-api-access-sb46c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:48:04.016248 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.016224    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "920dfec1-ebeb-42b4-9d69-1bf654bca46f" (UID: "920dfec1-ebeb-42b4-9d69-1bf654bca46f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:48:04.097160 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.097127    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:04.097160 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.097156    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:04.097160 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.097167    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:04.097428 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.097178    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb46c\" (UniqueName: \"kubernetes.io/projected/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kube-api-access-sb46c\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:04.097428 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.097193    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920dfec1-ebeb-42b4-9d69-1bf654bca46f-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:04.097428 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.097206    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920dfec1-ebeb-42b4-9d69-1bf654bca46f-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:04.097428 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.097218    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920dfec1-ebeb-42b4-9d69-1bf654bca46f-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:48:04.148434 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.148399    2582 generic.go:356] "Generic (PLEG): container finished" podID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerID="99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9" exitCode=0
Sep 18 22:48:04.148580 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.148469    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"
Sep 18 22:48:04.148580 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.148485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" event={"ID":"920dfec1-ebeb-42b4-9d69-1bf654bca46f","Type":"ContainerDied","Data":"99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9"}
Sep 18 22:48:04.148580 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.148574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw" event={"ID":"920dfec1-ebeb-42b4-9d69-1bf654bca46f","Type":"ContainerDied","Data":"691535c638d6b9ba71975f26f3cb42e6528acb132d917d76020c19a212abe53a"}
Sep 18 22:48:04.148686 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.148600    2582 scope.go:117] "RemoveContainer" containerID="99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9"
Sep 18 22:48:04.158267 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.158253    2582 scope.go:117] "RemoveContainer" containerID="36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07"
Sep 18 22:48:04.169236 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.169216    2582 scope.go:117] "RemoveContainer" containerID="99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9"
Sep 18 22:48:04.169550 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:48:04.169525    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9\": container with ID starting with 99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9 not found: ID does not exist" containerID="99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9"
Sep 18 22:48:04.169599 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.169562    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9"} err="failed to get container status \"99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9\": rpc error: code = NotFound desc = could not find container \"99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9\": container with ID starting with 99d6d80ab4539448132311b4c53d4087b2b47b85b5ff619b4407e8a41c20c4b9 not found: ID does not exist"
Sep 18 22:48:04.169599 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.169587    2582 scope.go:117] "RemoveContainer" containerID="36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07"
Sep 18 22:48:04.169846 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:48:04.169826    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07\": container with ID starting with 36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07 not found: ID does not exist" containerID="36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07"
Sep 18 22:48:04.169909 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.169853    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07"} err="failed to get container status \"36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07\": rpc error: code = NotFound desc = could not find container \"36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07\": container with ID starting with 36f9f587acb646ff3c8e690347cd7190ae9c7da2ebc5b32cbc78ce98b0aa5b07 not found: ID does not exist"
Sep 18 22:48:04.173187 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.173164    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"]
Sep 18 22:48:04.176770 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.176748    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-75f6996fb5-tpvrw"]
Sep 18 22:48:04.975253 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:04.975223    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" path="/var/lib/kubelet/pods/920dfec1-ebeb-42b4-9d69-1bf654bca46f/volumes"
Sep 18 22:48:41.039319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039280    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"]
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039671    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="storage-initializer"
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039689    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="storage-initializer"
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039728    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main"
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039737    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main"
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039748    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main"
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039756    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main"
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039853    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="920dfec1-ebeb-42b4-9d69-1bf654bca46f" containerName="main"
Sep 18 22:48:41.041887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.039872    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4692c9-68eb-4e76-92d6-8043b4298de9" containerName="main"
Sep 18 22:48:41.043289 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.043262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.045901 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.045878    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-9dgjn\""
Sep 18 22:48:41.046103 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.046087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-szxv4\""
Sep 18 22:48:41.046517 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.046495    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 22:48:41.046618 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.046529    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 18 22:48:41.046618 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.046505    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:48:41.046877 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.046859    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 22:48:41.046935 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.046885    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 22:48:41.058608 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.058573    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"]
Sep 18 22:48:41.107713 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.107678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.107713 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.107714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.107930 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.107738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81eae2c8-892e-4b7f-9624-825067b9be42-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.107930 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.107808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.107930 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.107877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.108073 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.107932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.108073 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.108017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eae2c8-892e-4b7f-9624-825067b9be42-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.108073 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.108034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chpgf\" (UniqueName: \"kubernetes.io/projected/81eae2c8-892e-4b7f-9624-825067b9be42-kube-api-access-chpgf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209393 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209595 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209595 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eae2c8-892e-4b7f-9624-825067b9be42-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209595 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chpgf\" (UniqueName: \"kubernetes.io/projected/81eae2c8-892e-4b7f-9624-825067b9be42-kube-api-access-chpgf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209780 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209780 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209780 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81eae2c8-892e-4b7f-9624-825067b9be42-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.209780 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.210031 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.209828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.210107 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.210081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.210173 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.210103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.210230 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.210165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.210450 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.210430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81eae2c8-892e-4b7f-9624-825067b9be42-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.211705 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.211682    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.211989 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.211971    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eae2c8-892e-4b7f-9624-825067b9be42-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.220660 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.220639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chpgf\" (UniqueName: \"kubernetes.io/projected/81eae2c8-892e-4b7f-9624-825067b9be42-kube-api-access-chpgf\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.354315 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.354228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:41.671419 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:41.671392    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"]
Sep 18 22:48:41.673117 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:48:41.673088    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81eae2c8_892e_4b7f_9624_825067b9be42.slice/crio-3bc0dd303c9f264813f41afadebe15ce5acdeb24bf39a3854b920f0b939c296d WatchSource:0}: Error finding container 3bc0dd303c9f264813f41afadebe15ce5acdeb24bf39a3854b920f0b939c296d: Status 404 returned error can't find the container with id 3bc0dd303c9f264813f41afadebe15ce5acdeb24bf39a3854b920f0b939c296d
Sep 18 22:48:42.354245 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:42.354201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerStarted","Data":"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"}
Sep 18 22:48:42.354663 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:42.354313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:42.354663 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:42.354341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerStarted","Data":"3bc0dd303c9f264813f41afadebe15ce5acdeb24bf39a3854b920f0b939c296d"}
Sep 18 22:48:43.363305 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:43.363268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerStarted","Data":"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"}
Sep 18 22:48:45.376312 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:45.376276    2582 generic.go:356] "Generic (PLEG): container finished" podID="81eae2c8-892e-4b7f-9624-825067b9be42" containerID="16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f" exitCode=0
Sep 18 22:48:45.376721 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:45.376332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerDied","Data":"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"}
Sep 18 22:48:46.385023 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:46.384982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerStarted","Data":"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"}
Sep 18 22:48:46.412081 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:46.412016    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podStartSLOduration=3.073925783 podStartE2EDuration="5.411981831s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.338056048s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:48:46.409141278 +0000 UTC m=+3483.442320173" watchObservedRunningTime="2026-09-18 22:48:46.411981831 +0000 UTC m=+3483.445160725"
Sep 18 22:48:51.354992 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:51.354924    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:51.355414 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:51.355026    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:48:51.356454 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:51.356412    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:48:51.369627 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:48:51.369603    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:49:01.355362 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:49:01.355255    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:49:11.355233 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:49:11.355185    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:49:21.355374 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:49:21.355316    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:49:31.355577 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:49:31.355524    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:49:41.354787 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:49:41.354739    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:49:51.355214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:49:51.355172    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:50:01.354851 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:01.354808    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:50:11.354925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:11.354873    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:50:21.354825 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:21.354775    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:50:31.355454 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:31.355338    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 22:50:41.364839 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:41.364802    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:50:41.377171 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:41.377150    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:50:43.105993 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.105939    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"]
Sep 18 22:50:43.110233 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.110209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.114244 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.114221    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-rbr4p\""
Sep 18 22:50:43.114568 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.114293    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 22:50:43.114752 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.114350    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 22:50:43.114752 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.114607    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 22:50:43.123126 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.123099    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"]
Sep 18 22:50:43.185294 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.185264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cb91fb-bd3a-4355-9201-3223e9725bc0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.185468 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.185312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5mt\" (UniqueName: \"kubernetes.io/projected/03cb91fb-bd3a-4355-9201-3223e9725bc0-kube-api-access-5l5mt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.185468 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.185375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.185468 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.185406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.185614 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.185504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.185614 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.185567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.286580 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.286541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5mt\" (UniqueName: \"kubernetes.io/projected/03cb91fb-bd3a-4355-9201-3223e9725bc0-kube-api-access-5l5mt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.286763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.286594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.286763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.286617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.286763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.286660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.286763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.286682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.286763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.286753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cb91fb-bd3a-4355-9201-3223e9725bc0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.287095 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.287072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.287159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.287094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.287159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.287133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.288784 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.288754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.289200 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.289180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cb91fb-bd3a-4355-9201-3223e9725bc0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.298719 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.298688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5mt\" (UniqueName: \"kubernetes.io/projected/03cb91fb-bd3a-4355-9201-3223e9725bc0-kube-api-access-5l5mt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.423425 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.423394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:43.700865 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:43.700838    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"]
Sep 18 22:50:43.702888 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:50:43.702860    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cb91fb_bd3a_4355_9201_3223e9725bc0.slice/crio-bcbcfc2e58454487a7ea9b36224ac3975f55dbdf5948e2dd8b55b534f6bf261a WatchSource:0}: Error finding container bcbcfc2e58454487a7ea9b36224ac3975f55dbdf5948e2dd8b55b534f6bf261a: Status 404 returned error can't find the container with id bcbcfc2e58454487a7ea9b36224ac3975f55dbdf5948e2dd8b55b534f6bf261a
Sep 18 22:50:44.058783 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:44.058682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" event={"ID":"03cb91fb-bd3a-4355-9201-3223e9725bc0","Type":"ContainerStarted","Data":"28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492"}
Sep 18 22:50:44.058783 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:44.058751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" event={"ID":"03cb91fb-bd3a-4355-9201-3223e9725bc0","Type":"ContainerStarted","Data":"bcbcfc2e58454487a7ea9b36224ac3975f55dbdf5948e2dd8b55b534f6bf261a"}
Sep 18 22:50:44.079219 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:44.079167    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" podStartSLOduration=1.079151446 podStartE2EDuration="1.079151446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:50:44.076251621 +0000 UTC m=+3601.109430515" watchObservedRunningTime="2026-09-18 22:50:44.079151446 +0000 UTC m=+3601.112330339"
Sep 18 22:50:44.423925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:44.423882    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:45.309550 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:45.309520    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:50:45.316021 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:45.315998    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:50:45.321377 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:45.321359    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:50:45.327329 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:45.327302    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:50:53.423686 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:53.423655    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:53.436460 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:53.436423    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:50:54.125806 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:50:54.125773    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:51:11.733756 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:11.733716    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"]
Sep 18 22:51:21.474369 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.474335    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"]
Sep 18 22:51:21.484225 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:51:21.484190    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 22:51:21.484225 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:51:21.484225 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:51:21.484225 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492" cmd=["/bin/sleep","15"]
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.484249    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" message="" err=<
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:51:21.484261    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" podUID="03cb91fb-bd3a-4355-9201-3223e9725bc0" containerName="main" containerID="cri-o://28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492"
Sep 18 22:51:21.484466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.484308    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" podUID="03cb91fb-bd3a-4355-9201-3223e9725bc0" containerName="main" containerID="cri-o://28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492" gracePeriod=60
Sep 18 22:51:21.837332 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.837307    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:51:21.937641 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.937600    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cb91fb-bd3a-4355-9201-3223e9725bc0-tls-certs\") pod \"03cb91fb-bd3a-4355-9201-3223e9725bc0\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") "
Sep 18 22:51:21.937641 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.937650    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-dshm\") pod \"03cb91fb-bd3a-4355-9201-3223e9725bc0\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") "
Sep 18 22:51:21.938000 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.937679    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-model-cache\") pod \"03cb91fb-bd3a-4355-9201-3223e9725bc0\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") "
Sep 18 22:51:21.938000 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.937722    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-tmp-dir\") pod \"03cb91fb-bd3a-4355-9201-3223e9725bc0\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") "
Sep 18 22:51:21.938000 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.937761    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l5mt\" (UniqueName: \"kubernetes.io/projected/03cb91fb-bd3a-4355-9201-3223e9725bc0-kube-api-access-5l5mt\") pod \"03cb91fb-bd3a-4355-9201-3223e9725bc0\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") "
Sep 18 22:51:21.938000 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.937796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-home\") pod \"03cb91fb-bd3a-4355-9201-3223e9725bc0\" (UID: \"03cb91fb-bd3a-4355-9201-3223e9725bc0\") "
Sep 18 22:51:21.938179 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.938005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "03cb91fb-bd3a-4355-9201-3223e9725bc0" (UID: "03cb91fb-bd3a-4355-9201-3223e9725bc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:21.938179 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.938018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03cb91fb-bd3a-4355-9201-3223e9725bc0" (UID: "03cb91fb-bd3a-4355-9201-3223e9725bc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:21.938179 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.938126    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:21.938179 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.938134    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-home" (OuterVolumeSpecName: "home") pod "03cb91fb-bd3a-4355-9201-3223e9725bc0" (UID: "03cb91fb-bd3a-4355-9201-3223e9725bc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:21.938179 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.938144    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:21.939830 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.939801    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cb91fb-bd3a-4355-9201-3223e9725bc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03cb91fb-bd3a-4355-9201-3223e9725bc0" (UID: "03cb91fb-bd3a-4355-9201-3223e9725bc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:51:21.940010 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.939867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cb91fb-bd3a-4355-9201-3223e9725bc0-kube-api-access-5l5mt" (OuterVolumeSpecName: "kube-api-access-5l5mt") pod "03cb91fb-bd3a-4355-9201-3223e9725bc0" (UID: "03cb91fb-bd3a-4355-9201-3223e9725bc0"). InnerVolumeSpecName "kube-api-access-5l5mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:21.940010 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:21.939881    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-dshm" (OuterVolumeSpecName: "dshm") pod "03cb91fb-bd3a-4355-9201-3223e9725bc0" (UID: "03cb91fb-bd3a-4355-9201-3223e9725bc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:22.039277 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.039198    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03cb91fb-bd3a-4355-9201-3223e9725bc0-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:22.039277 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.039224    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:22.039277 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.039235    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l5mt\" (UniqueName: \"kubernetes.io/projected/03cb91fb-bd3a-4355-9201-3223e9725bc0-kube-api-access-5l5mt\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:22.039277 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.039247    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03cb91fb-bd3a-4355-9201-3223e9725bc0-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:22.274222 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.274185    2582 generic.go:356] "Generic (PLEG): container finished" podID="03cb91fb-bd3a-4355-9201-3223e9725bc0" containerID="28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492" exitCode=0
Sep 18 22:51:22.274394 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.274248    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"
Sep 18 22:51:22.274394 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.274261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" event={"ID":"03cb91fb-bd3a-4355-9201-3223e9725bc0","Type":"ContainerDied","Data":"28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492"}
Sep 18 22:51:22.274394 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.274334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b" event={"ID":"03cb91fb-bd3a-4355-9201-3223e9725bc0","Type":"ContainerDied","Data":"bcbcfc2e58454487a7ea9b36224ac3975f55dbdf5948e2dd8b55b534f6bf261a"}
Sep 18 22:51:22.274394 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.274363    2582 scope.go:117] "RemoveContainer" containerID="28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492"
Sep 18 22:51:22.284594 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.284577    2582 scope.go:117] "RemoveContainer" containerID="28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492"
Sep 18 22:51:22.284850 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:51:22.284831    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492\": container with ID starting with 28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492 not found: ID does not exist" containerID="28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492"
Sep 18 22:51:22.284896 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.284857    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492"} err="failed to get container status \"28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492\": rpc error: code = NotFound desc = could not find container \"28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492\": container with ID starting with 28bf82d724712c3f0b6951208b124197c5c721ed6a604edb8aa945bb6e9c7492 not found: ID does not exist"
Sep 18 22:51:22.298158 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.298094    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"]
Sep 18 22:51:22.310609 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.310589    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-sxz8b"]
Sep 18 22:51:22.981981 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:22.981928    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03cb91fb-bd3a-4355-9201-3223e9725bc0" path="/var/lib/kubelet/pods/03cb91fb-bd3a-4355-9201-3223e9725bc0/volumes"
Sep 18 22:51:26.758467 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:26.758422    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main" containerID="cri-o://347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84" gracePeriod=45
Sep 18 22:51:27.375385 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.375345    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="llm-d-routing-sidecar" containerID="cri-o://bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4" gracePeriod=45
Sep 18 22:51:27.633904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.633877    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:51:27.690280 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690251    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-kserve-provision-location\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690292    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-model-cache\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eae2c8-892e-4b7f-9624-825067b9be42-tls-certs\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690366    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81eae2c8-892e-4b7f-9624-825067b9be42-cabundle-cert\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-tmp-dir\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690451 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690423    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-home\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690733 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690624    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-model-cache" (OuterVolumeSpecName: "model-cache") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.690733 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690629    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-dshm\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690733 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chpgf\" (UniqueName: \"kubernetes.io/projected/81eae2c8-892e-4b7f-9624-825067b9be42-kube-api-access-chpgf\") pod \"81eae2c8-892e-4b7f-9624-825067b9be42\" (UID: \"81eae2c8-892e-4b7f-9624-825067b9be42\") "
Sep 18 22:51:27.690890 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.690819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81eae2c8-892e-4b7f-9624-825067b9be42-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:51:27.691088 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.691067    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81eae2c8-892e-4b7f-9624-825067b9be42-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.691207 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.691091    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.691207 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.691099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-home" (OuterVolumeSpecName: "home") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.692785 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.692750    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-dshm" (OuterVolumeSpecName: "dshm") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.692904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.692874    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81eae2c8-892e-4b7f-9624-825067b9be42-kube-api-access-chpgf" (OuterVolumeSpecName: "kube-api-access-chpgf") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "kube-api-access-chpgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:27.693059 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.692980    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81eae2c8-892e-4b7f-9624-825067b9be42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:51:27.705138 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.705114    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.710146 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.710124    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81eae2c8-892e-4b7f-9624-825067b9be42" (UID: "81eae2c8-892e-4b7f-9624-825067b9be42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:27.791644 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.791618    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eae2c8-892e-4b7f-9624-825067b9be42-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.791644 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.791642    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.792060 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.791653    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.792060 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.791664    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.792060 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.791672    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chpgf\" (UniqueName: \"kubernetes.io/projected/81eae2c8-892e-4b7f-9624-825067b9be42-kube-api-access-chpgf\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:27.792060 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:27.791681    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81eae2c8-892e-4b7f-9624-825067b9be42-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:51:28.310818 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.310784    2582 generic.go:356] "Generic (PLEG): container finished" podID="81eae2c8-892e-4b7f-9624-825067b9be42" containerID="347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84" exitCode=0
Sep 18 22:51:28.310818 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.310811    2582 generic.go:356] "Generic (PLEG): container finished" podID="81eae2c8-892e-4b7f-9624-825067b9be42" containerID="bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4" exitCode=0
Sep 18 22:51:28.311078 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.310840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerDied","Data":"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"}
Sep 18 22:51:28.311078 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.310854    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"
Sep 18 22:51:28.311078 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.310886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerDied","Data":"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"}
Sep 18 22:51:28.311078 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.310903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9" event={"ID":"81eae2c8-892e-4b7f-9624-825067b9be42","Type":"ContainerDied","Data":"3bc0dd303c9f264813f41afadebe15ce5acdeb24bf39a3854b920f0b939c296d"}
Sep 18 22:51:28.311078 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.310924    2582 scope.go:117] "RemoveContainer" containerID="347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"
Sep 18 22:51:28.322616 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.322585    2582 scope.go:117] "RemoveContainer" containerID="16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"
Sep 18 22:51:28.332145 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.332124    2582 scope.go:117] "RemoveContainer" containerID="bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"
Sep 18 22:51:28.341175 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.341133    2582 scope.go:117] "RemoveContainer" containerID="347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"
Sep 18 22:51:28.341467 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:51:28.341444    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84\": container with ID starting with 347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84 not found: ID does not exist" containerID="347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"
Sep 18 22:51:28.341533 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.341475    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"} err="failed to get container status \"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84\": rpc error: code = NotFound desc = could not find container \"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84\": container with ID starting with 347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84 not found: ID does not exist"
Sep 18 22:51:28.341533 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.341492    2582 scope.go:117] "RemoveContainer" containerID="16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"
Sep 18 22:51:28.341837 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:51:28.341807    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f\": container with ID starting with 16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f not found: ID does not exist" containerID="16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"
Sep 18 22:51:28.341892 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.341842    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"} err="failed to get container status \"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f\": rpc error: code = NotFound desc = could not find container \"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f\": container with ID starting with 16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f not found: ID does not exist"
Sep 18 22:51:28.341892 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.341859    2582 scope.go:117] "RemoveContainer" containerID="bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"
Sep 18 22:51:28.342006 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.341918    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"]
Sep 18 22:51:28.342139 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:51:28.342121    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4\": container with ID starting with bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4 not found: ID does not exist" containerID="bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"
Sep 18 22:51:28.342177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.342145    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"} err="failed to get container status \"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4\": rpc error: code = NotFound desc = could not find container \"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4\": container with ID starting with bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4 not found: ID does not exist"
Sep 18 22:51:28.342177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.342160    2582 scope.go:117] "RemoveContainer" containerID="347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"
Sep 18 22:51:28.342404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.342387    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84"} err="failed to get container status \"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84\": rpc error: code = NotFound desc = could not find container \"347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84\": container with ID starting with 347d7d48f0219c0a6fbb0698b866affcfe54d69b930836c8b4505ff470198b84 not found: ID does not exist"
Sep 18 22:51:28.342404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.342403    2582 scope.go:117] "RemoveContainer" containerID="16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"
Sep 18 22:51:28.342646 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.342626    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f"} err="failed to get container status \"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f\": rpc error: code = NotFound desc = could not find container \"16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f\": container with ID starting with 16b9e16e60a233457d95f1a0f8945bb28d5fafa8234c7606eef2864d325ee59f not found: ID does not exist"
Sep 18 22:51:28.342688 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.342646    2582 scope.go:117] "RemoveContainer" containerID="bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"
Sep 18 22:51:28.342850 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.342834    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4"} err="failed to get container status \"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4\": rpc error: code = NotFound desc = could not find container \"bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4\": container with ID starting with bc089a5177fa29addfc89f285afe4ae3ae6a398f6925476a527ada40e39fbbd4 not found: ID does not exist"
Sep 18 22:51:28.348724 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.348700    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f778797b-nw9r9"]
Sep 18 22:51:28.974518 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:51:28.974478    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" path="/var/lib/kubelet/pods/81eae2c8-892e-4b7f-9624-825067b9be42/volumes"
Sep 18 22:52:14.740332 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740296    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"]
Sep 18 22:52:14.740846 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740829    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="llm-d-routing-sidecar"
Sep 18 22:52:14.740925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740848    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="llm-d-routing-sidecar"
Sep 18 22:52:14.740925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740864    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="storage-initializer"
Sep 18 22:52:14.740925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740872    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="storage-initializer"
Sep 18 22:52:14.740925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740885    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03cb91fb-bd3a-4355-9201-3223e9725bc0" containerName="main"
Sep 18 22:52:14.740925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740894    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03cb91fb-bd3a-4355-9201-3223e9725bc0" containerName="main"
Sep 18 22:52:14.740925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740907    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main"
Sep 18 22:52:14.740925 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.740915    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main"
Sep 18 22:52:14.741305 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.741049    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="main"
Sep 18 22:52:14.741305 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.741064    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03cb91fb-bd3a-4355-9201-3223e9725bc0" containerName="main"
Sep 18 22:52:14.741305 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.741079    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="81eae2c8-892e-4b7f-9624-825067b9be42" containerName="llm-d-routing-sidecar"
Sep 18 22:52:14.746100 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.746078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.748797 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.748775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-bcvzt\""
Sep 18 22:52:14.748797 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.748791    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 22:52:14.749006 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.748776    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-2cbb6\""
Sep 18 22:52:14.749006 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.748777    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 22:52:14.749006 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.748779    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 22:52:14.749113 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.749062    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 22:52:14.749628 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.749614    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:52:14.758764 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.758739    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"]
Sep 18 22:52:14.775314 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.775483 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhn9z\" (UniqueName: \"kubernetes.io/projected/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kube-api-access-dhn9z\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.775483 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.775669 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-home\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.775669 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-dshm\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.775669 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.775789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.775789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.775709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876579 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876579 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876579    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876807 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876807 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876807 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhn9z\" (UniqueName: \"kubernetes.io/projected/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kube-api-access-dhn9z\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876807 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876807 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-home\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.876807 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.876798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-dshm\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.877159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.877022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.877159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.877075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.877273 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.877161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.877336 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.877304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-home\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.877476 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.877456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.879069 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.879047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-dshm\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.879165 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.879136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:14.886236 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:14.886211    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhn9z\" (UniqueName: \"kubernetes.io/projected/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kube-api-access-dhn9z\") pod \"custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:15.055608 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:15.055518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:15.340847 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:15.340768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"]
Sep 18 22:52:15.344106 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:52:15.344078    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd34e6fa_72cb_4d36_9700_c6e4f7e61a6b.slice/crio-c138966b4daa50d5dd7476a72e4cbdcfdb4178589e1ad40f31b039eb13919e7a WatchSource:0}: Error finding container c138966b4daa50d5dd7476a72e4cbdcfdb4178589e1ad40f31b039eb13919e7a: Status 404 returned error can't find the container with id c138966b4daa50d5dd7476a72e4cbdcfdb4178589e1ad40f31b039eb13919e7a
Sep 18 22:52:15.564588 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:15.564547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerStarted","Data":"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"}
Sep 18 22:52:15.564763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:15.564635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerStarted","Data":"c138966b4daa50d5dd7476a72e4cbdcfdb4178589e1ad40f31b039eb13919e7a"}
Sep 18 22:52:15.564763 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:15.564664    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:16.571634 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:16.571596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerStarted","Data":"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"}
Sep 18 22:52:18.586197 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:18.586161    2582 generic.go:356] "Generic (PLEG): container finished" podID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerID="4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367" exitCode=0
Sep 18 22:52:18.586594 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:18.586211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerDied","Data":"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"}
Sep 18 22:52:19.594193 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:19.594158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerStarted","Data":"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"}
Sep 18 22:52:19.620839 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:19.620777    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podStartSLOduration=3.408437181 podStartE2EDuration="5.620761324s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.212324143s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:52:19.619477119 +0000 UTC m=+3696.652656013" watchObservedRunningTime="2026-09-18 22:52:19.620761324 +0000 UTC m=+3696.653940218"
Sep 18 22:52:25.056194 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:25.056142    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:25.056828 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:25.056224    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:25.057797 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:25.057759    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:52:25.090339 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:25.090308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:52:35.056544 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:35.056486    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:52:45.056906 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:45.056828    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:52:55.056584 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:52:55.056540    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:53:05.056633 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:05.056571    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:53:15.056397 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:15.056351    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:53:25.056497 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:25.056440    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:53:30.530904 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.530812    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"]
Sep 18 22:53:30.534720 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.534699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.537574 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.537546    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 22:53:30.537706 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.537558    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-rfwtv\""
Sep 18 22:53:30.537706 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.537558    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 22:53:30.537706 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.537559    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 22:53:30.537706 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.537643    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:53:30.538079 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.538064    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 18 22:53:30.556630 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.556607    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"]
Sep 18 22:53:30.642506 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwhr\" (UniqueName: \"kubernetes.io/projected/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kube-api-access-ptwhr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.642684 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.642684 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.642684 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.642684 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.642923 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.642923 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.642923 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.642821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.744252 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.744455 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.744455 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwhr\" (UniqueName: \"kubernetes.io/projected/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kube-api-access-ptwhr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.744707 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.744810 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.744810 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744757    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.744810 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.745010 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.745010 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.744883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.745117 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.745078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.745359 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.745336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.745519 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.745436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.745628 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.745584    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.747588 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.747566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.747996 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.747972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.758039 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.758012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwhr\" (UniqueName: \"kubernetes.io/projected/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kube-api-access-ptwhr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:30.845379 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:30.845276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:31.141632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:31.141599    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"]
Sep 18 22:53:31.143265 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:53:31.143236    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe67593b_0ee0_4c0e_8d9a_54ff1ee7dc5c.slice/crio-eea362f062cacfbbda032f6ce5e2307b45868210c2f4833fe257e94683455f09 WatchSource:0}: Error finding container eea362f062cacfbbda032f6ce5e2307b45868210c2f4833fe257e94683455f09: Status 404 returned error can't find the container with id eea362f062cacfbbda032f6ce5e2307b45868210c2f4833fe257e94683455f09
Sep 18 22:53:32.013086 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:32.013049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" event={"ID":"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c","Type":"ContainerStarted","Data":"fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa"}
Sep 18 22:53:32.013666 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:32.013101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" event={"ID":"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c","Type":"ContainerStarted","Data":"eea362f062cacfbbda032f6ce5e2307b45868210c2f4833fe257e94683455f09"}
Sep 18 22:53:34.027300 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:34.027268    2582 generic.go:356] "Generic (PLEG): container finished" podID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerID="fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa" exitCode=0
Sep 18 22:53:34.027783 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:34.027339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" event={"ID":"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c","Type":"ContainerDied","Data":"fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa"}
Sep 18 22:53:35.036288 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:35.036253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" event={"ID":"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c","Type":"ContainerStarted","Data":"9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615"}
Sep 18 22:53:35.056498 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:35.056454    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:53:35.067182 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:35.067118    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podStartSLOduration=2.29080265 podStartE2EDuration="5.067097764s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.776295114s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:53:35.062235027 +0000 UTC m=+3772.095413922" watchObservedRunningTime="2026-09-18 22:53:35.067097764 +0000 UTC m=+3772.100276660"
Sep 18 22:53:35.846065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:35.846018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:40.845620 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:40.845555    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:53:40.847830 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:40.847793    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:53:45.057123 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:45.056996    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:53:50.846214 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:50.846162    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:53:55.056159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:53:55.056102    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:54:00.846636 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:00.846588    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:54:05.056338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:05.056281    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 22:54:10.846469 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:10.846413    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:54:15.066499 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:15.066459    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:54:15.082641 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:15.082612    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:54:20.846466 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:20.846413    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:54:24.544819 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:24.544779    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"]
Sep 18 22:54:30.846244 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:30.846188    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:54:39.568364 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:39.568322    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main" containerID="cri-o://7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d" gracePeriod=45
Sep 18 22:54:40.185571 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.185504    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="llm-d-routing-sidecar" containerID="cri-o://ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f" gracePeriod=45
Sep 18 22:54:40.429856 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.429832    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:54:40.450184 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.450152    2582 generic.go:356] "Generic (PLEG): container finished" podID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerID="7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d" exitCode=0
Sep 18 22:54:40.450184 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.450182    2582 generic.go:356] "Generic (PLEG): container finished" podID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerID="ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f" exitCode=0
Sep 18 22:54:40.450361 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.450258    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"
Sep 18 22:54:40.450361 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.450282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerDied","Data":"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"}
Sep 18 22:54:40.450361 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.450338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerDied","Data":"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"}
Sep 18 22:54:40.450463 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.450364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f" event={"ID":"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b","Type":"ContainerDied","Data":"c138966b4daa50d5dd7476a72e4cbdcfdb4178589e1ad40f31b039eb13919e7a"}
Sep 18 22:54:40.450463 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.450387    2582 scope.go:117] "RemoveContainer" containerID="7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"
Sep 18 22:54:40.463235 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.463208    2582 scope.go:117] "RemoveContainer" containerID="4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"
Sep 18 22:54:40.474305 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.474279    2582 scope.go:117] "RemoveContainer" containerID="ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"
Sep 18 22:54:40.482565 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.482544    2582 scope.go:117] "RemoveContainer" containerID="7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"
Sep 18 22:54:40.482937 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:54:40.482917    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d\": container with ID starting with 7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d not found: ID does not exist" containerID="7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"
Sep 18 22:54:40.483025 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.482971    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"} err="failed to get container status \"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d\": rpc error: code = NotFound desc = could not find container \"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d\": container with ID starting with 7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d not found: ID does not exist"
Sep 18 22:54:40.483025 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.482996    2582 scope.go:117] "RemoveContainer" containerID="4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"
Sep 18 22:54:40.483294 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:54:40.483275    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367\": container with ID starting with 4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367 not found: ID does not exist" containerID="4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"
Sep 18 22:54:40.483362 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.483302    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"} err="failed to get container status \"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367\": rpc error: code = NotFound desc = could not find container \"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367\": container with ID starting with 4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367 not found: ID does not exist"
Sep 18 22:54:40.483362 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.483317    2582 scope.go:117] "RemoveContainer" containerID="ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"
Sep 18 22:54:40.483587 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:54:40.483569    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f\": container with ID starting with ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f not found: ID does not exist" containerID="ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"
Sep 18 22:54:40.483648 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.483592    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"} err="failed to get container status \"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f\": rpc error: code = NotFound desc = could not find container \"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f\": container with ID starting with ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f not found: ID does not exist"
Sep 18 22:54:40.483648 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.483608    2582 scope.go:117] "RemoveContainer" containerID="7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"
Sep 18 22:54:40.483872 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.483849    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d"} err="failed to get container status \"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d\": rpc error: code = NotFound desc = could not find container \"7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d\": container with ID starting with 7c801c367ff3c6bc182f834ae88bcd4b8e72c46e29f52a6ca51b72f32943b80d not found: ID does not exist"
Sep 18 22:54:40.483975 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.483875    2582 scope.go:117] "RemoveContainer" containerID="4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"
Sep 18 22:54:40.484319 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.484293    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367"} err="failed to get container status \"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367\": rpc error: code = NotFound desc = could not find container \"4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367\": container with ID starting with 4127cbed33d9768c7e3d01f29880b5c0012416bde9e8ca0a9633b354c6b46367 not found: ID does not exist"
Sep 18 22:54:40.484389 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.484320    2582 scope.go:117] "RemoveContainer" containerID="ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"
Sep 18 22:54:40.484587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.484560    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f"} err="failed to get container status \"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f\": rpc error: code = NotFound desc = could not find container \"ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f\": container with ID starting with ac06f52de7286396c32da7bdc34be52fe33e0860c99481b2c3f764f822348d6f not found: ID does not exist"
Sep 18 22:54:40.589165 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589133    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-cabundle-cert\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589231    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tmp-dir\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589276    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-model-cache\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589300    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-home\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589327    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-dshm\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kserve-provision-location\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589413    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tls-certs\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhn9z\" (UniqueName: \"kubernetes.io/projected/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kube-api-access-dhn9z\") pod \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\" (UID: \"cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b\") "
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589524    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.589632 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589540    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:54:40.590177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589821    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.590177 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.589841    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.590308 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.590283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-home" (OuterVolumeSpecName: "home") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.591930 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.591894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:54:40.592261 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.592153    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-dshm" (OuterVolumeSpecName: "dshm") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.592329 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.592266    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kube-api-access-dhn9z" (OuterVolumeSpecName: "kube-api-access-dhn9z") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "kube-api-access-dhn9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:54:40.601385 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.601351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.606351 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.606324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" (UID: "cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:40.690509 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.690472    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.690509 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.690505    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.690509 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.690515    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.690808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.690523    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.690808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.690532    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.690808 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.690540    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhn9z\" (UniqueName: \"kubernetes.io/projected/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b-kube-api-access-dhn9z\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:54:40.779706 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.779675    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"]
Sep 18 22:54:40.784056 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.784027    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-785dc578d5-dcw9f"]
Sep 18 22:54:40.845969 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.845914    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:54:40.975353 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:40.975306    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" path="/var/lib/kubelet/pods/cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b/volumes"
Sep 18 22:54:50.856228 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:50.856195    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:54:50.863888 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:54:50.863865    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:55:19.749901 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:19.749858    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"]
Sep 18 22:55:34.772614 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:34.772571    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" containerID="cri-o://9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615" gracePeriod=45
Sep 18 22:55:34.857379 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:34.857331    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 22:55:35.418787 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.418757    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:55:35.562735 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.562648    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-cabundle-cert\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.562735 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.562696    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tmp-dir\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.562735 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.562722    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-home\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.563077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.562818    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tls-certs\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.563077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.562849    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptwhr\" (UniqueName: \"kubernetes.io/projected/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kube-api-access-ptwhr\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.563077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.562885    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-dshm\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.563077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.562925    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-model-cache\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.563077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.563072    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kserve-provision-location\") pod \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\" (UID: \"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c\") "
Sep 18 22:55:35.563343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.563161    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:55:35.563343 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.563219    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-home" (OuterVolumeSpecName: "home") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:55:35.563458 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.563436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:55:35.563536 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.563517    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.563593 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.563541    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.563593 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.563555    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.565747 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.565691    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-dshm" (OuterVolumeSpecName: "dshm") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:55:35.565874 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.565767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:55:35.565874 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.565784    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kube-api-access-ptwhr" (OuterVolumeSpecName: "kube-api-access-ptwhr") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "kube-api-access-ptwhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:55:35.576624 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.576595    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:55:35.585155 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.585131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" (UID: "be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:55:35.664407 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.664374    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.664407 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.664404    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwhr\" (UniqueName: \"kubernetes.io/projected/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kube-api-access-ptwhr\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.664605 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.664415    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.664605 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.664425    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.664605 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.664433    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:55:35.760492 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.760456    2582 generic.go:356] "Generic (PLEG): container finished" podID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerID="9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615" exitCode=0
Sep 18 22:55:35.760687 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.760519    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"
Sep 18 22:55:35.760687 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.760540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" event={"ID":"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c","Type":"ContainerDied","Data":"9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615"}
Sep 18 22:55:35.760687 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.760599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq" event={"ID":"be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c","Type":"ContainerDied","Data":"eea362f062cacfbbda032f6ce5e2307b45868210c2f4833fe257e94683455f09"}
Sep 18 22:55:35.760687 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.760623    2582 scope.go:117] "RemoveContainer" containerID="9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615"
Sep 18 22:55:35.770329 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.770312    2582 scope.go:117] "RemoveContainer" containerID="fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa"
Sep 18 22:55:35.778915 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.778739    2582 scope.go:117] "RemoveContainer" containerID="9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615"
Sep 18 22:55:35.779180 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:55:35.779041    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615\": container with ID starting with 9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615 not found: ID does not exist" containerID="9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615"
Sep 18 22:55:35.779180 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.779075    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615"} err="failed to get container status \"9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615\": rpc error: code = NotFound desc = could not find container \"9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615\": container with ID starting with 9c011d9ec89ee68d443c2f9d8a92fb26805482d0b80e9ee3d4dd825f58007615 not found: ID does not exist"
Sep 18 22:55:35.779180 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.779093    2582 scope.go:117] "RemoveContainer" containerID="fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa"
Sep 18 22:55:35.779344 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:55:35.779329    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa\": container with ID starting with fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa not found: ID does not exist" containerID="fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa"
Sep 18 22:55:35.779385 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.779347    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa"} err="failed to get container status \"fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa\": rpc error: code = NotFound desc = could not find container \"fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa\": container with ID starting with fe409f56d5009a435b67863494a4d2c5a64d39c1ced6922d85c247be3f626afa not found: ID does not exist"
Sep 18 22:55:35.784182 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.784158    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"]
Sep 18 22:55:35.789648 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:35.789627    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-849d4db586-45xvq"]
Sep 18 22:55:36.974710 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:36.974675    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" path="/var/lib/kubelet/pods/be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c/volumes"
Sep 18 22:55:45.347879 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:45.347846    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:55:45.353873 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:45.353841    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:55:45.361190 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:45.361171    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 22:55:45.366447 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:45.366428    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 22:55:48.423235 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423197    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"]
Sep 18 22:55:48.423621 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423562    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main"
Sep 18 22:55:48.423621 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423573    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main"
Sep 18 22:55:48.423621 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423584    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="llm-d-routing-sidecar"
Sep 18 22:55:48.423621 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423591    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="llm-d-routing-sidecar"
Sep 18 22:55:48.423621 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423608    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main"
Sep 18 22:55:48.423621 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423614    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main"
Sep 18 22:55:48.423621 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423624    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="storage-initializer"
Sep 18 22:55:48.424050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423630    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="storage-initializer"
Sep 18 22:55:48.424050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423636    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="storage-initializer"
Sep 18 22:55:48.424050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423642    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="storage-initializer"
Sep 18 22:55:48.424050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423700    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="main"
Sep 18 22:55:48.424050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423709    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="be67593b-0ee0-4c0e-8d9a-54ff1ee7dc5c" containerName="main"
Sep 18 22:55:48.424050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.423718    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd34e6fa-72cb-4d36-9700-c6e4f7e61a6b" containerName="llm-d-routing-sidecar"
Sep 18 22:55:48.428491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.428470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.430834 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.430799    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 22:55:48.430975 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.430958    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 22:55:48.431056 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.430937    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-6rnh4\""
Sep 18 22:55:48.431160 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.431141    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-6mhsb\""
Sep 18 22:55:48.431218 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.431204    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:55:48.431399 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.431381    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 22:55:48.431728 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.431705    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 22:55:48.440194 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.440165    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"]
Sep 18 22:55:48.580789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.580739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d174743-5feb-424d-aaa8-9896eda12d8c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.580789 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.580788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-home\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.581099 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.580868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-model-cache\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.581099 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.580899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d174743-5feb-424d-aaa8-9896eda12d8c-tls-certs\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.581099 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.580963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4zh\" (UniqueName: \"kubernetes.io/projected/0d174743-5feb-424d-aaa8-9896eda12d8c-kube-api-access-5r4zh\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.581099 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.581024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.581099 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.581077    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.581306 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.581135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-dshm\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682412 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-model-cache\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682412 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d174743-5feb-424d-aaa8-9896eda12d8c-tls-certs\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682412 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4zh\" (UniqueName: \"kubernetes.io/projected/0d174743-5feb-424d-aaa8-9896eda12d8c-kube-api-access-5r4zh\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-dshm\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d174743-5feb-424d-aaa8-9896eda12d8c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.682714 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-home\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.683019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.683019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-model-cache\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.683019 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.682872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.683203 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.683018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-home\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.683327 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.683305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d174743-5feb-424d-aaa8-9896eda12d8c-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.684894 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.684870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-dshm\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.685116 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.685097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d174743-5feb-424d-aaa8-9896eda12d8c-tls-certs\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.690529 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.690509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4zh\" (UniqueName: \"kubernetes.io/projected/0d174743-5feb-424d-aaa8-9896eda12d8c-kube-api-access-5r4zh\") pod \"router-with-refs-pd-test-kserve-5fb5c479f5-smfn2\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:48.739065 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:48.739025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:49.025230 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:49.025196    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"]
Sep 18 22:55:49.025974 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:55:49.025934    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d174743_5feb_424d_aaa8_9896eda12d8c.slice/crio-d3d34b3fda1a012db7f1c81cb137a9e8f0f91878fb323b8ea88f6ac65d57f998 WatchSource:0}: Error finding container d3d34b3fda1a012db7f1c81cb137a9e8f0f91878fb323b8ea88f6ac65d57f998: Status 404 returned error can't find the container with id d3d34b3fda1a012db7f1c81cb137a9e8f0f91878fb323b8ea88f6ac65d57f998
Sep 18 22:55:49.839669 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:49.839628    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerStarted","Data":"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"}
Sep 18 22:55:49.840094 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:49.839729    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:49.840094 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:49.839749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerStarted","Data":"d3d34b3fda1a012db7f1c81cb137a9e8f0f91878fb323b8ea88f6ac65d57f998"}
Sep 18 22:55:50.847433 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:50.847391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerStarted","Data":"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"}
Sep 18 22:55:52.862357 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:52.862315    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerID="abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a" exitCode=0
Sep 18 22:55:52.862725 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:52.862377    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerDied","Data":"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"}
Sep 18 22:55:53.870171 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:53.870137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerStarted","Data":"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"}
Sep 18 22:55:53.895876 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:53.895816    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podStartSLOduration=3.641357432 podStartE2EDuration="5.895799674s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.254442242s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:55:53.89256207 +0000 UTC m=+3910.925740964" watchObservedRunningTime="2026-09-18 22:55:53.895799674 +0000 UTC m=+3910.928978569"
Sep 18 22:55:58.739551 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:58.739513    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:58.740021 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:58.739584    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:55:58.741066 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:55:58.741035    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:56:05.949590 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:05.949550    2582 generic.go:356] "Generic (PLEG): container finished" podID="08b849cf-a5b7-4831-9d0d-347e2239b1eb" containerID="91efa9855cda08813b8dd989bdd96144746e20e46271fed781575cfe3b5576c1" exitCode=1
Sep 18 22:56:05.950211 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:05.949630    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275" event={"ID":"08b849cf-a5b7-4831-9d0d-347e2239b1eb","Type":"ContainerDied","Data":"91efa9855cda08813b8dd989bdd96144746e20e46271fed781575cfe3b5576c1"}
Sep 18 22:56:05.950211 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:05.950036    2582 scope.go:117] "RemoveContainer" containerID="91efa9855cda08813b8dd989bdd96144746e20e46271fed781575cfe3b5576c1"
Sep 18 22:56:05.950347 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:05.950328    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:56:06.690267 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:06.690227    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 22:56:06.957238 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:06.957197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275" event={"ID":"08b849cf-a5b7-4831-9d0d-347e2239b1eb","Type":"ContainerStarted","Data":"12811af590574125e7ab5552a8ef1d4493f96d590cd8c40d20094acab7c6f98f"}
Sep 18 22:56:06.957720 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:06.957329    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 22:56:08.739614 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:08.739565    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:56:08.753366 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:08.753332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:56:14.134044 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.134005    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"]
Sep 18 22:56:14.144386 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.144360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.148045 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.148017    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 22:56:14.148314 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.148294    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 22:56:14.148407 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.148298    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 18 22:56:14.148407 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.148397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:56:14.148524 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.148403    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 22:56:14.148524 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.148491    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-4dm7n\""
Sep 18 22:56:14.156842 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.156817    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"]
Sep 18 22:56:14.203584 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.203584 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.203800 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0390769c-f94f-4cbe-878b-fd0942de55ff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.203800 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0390769c-f94f-4cbe-878b-fd0942de55ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.203800 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.203800 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.203800 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jgw9\" (UniqueName: \"kubernetes.io/projected/0390769c-f94f-4cbe-878b-fd0942de55ff-kube-api-access-2jgw9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.204017 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.203890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305266 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305437 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305437 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgw9\" (UniqueName: \"kubernetes.io/projected/0390769c-f94f-4cbe-878b-fd0942de55ff-kube-api-access-2jgw9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305437 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305437 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305648 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305648 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0390769c-f94f-4cbe-878b-fd0942de55ff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305648 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0390769c-f94f-4cbe-878b-fd0942de55ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305802 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305727    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.305909 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.305852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.306084 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.306053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.306155 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.306133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.306316 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.306298    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0390769c-f94f-4cbe-878b-fd0942de55ff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.308076 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.308057    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0390769c-f94f-4cbe-878b-fd0942de55ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.308194 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.308131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.313342 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.313321    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jgw9\" (UniqueName: \"kubernetes.io/projected/0390769c-f94f-4cbe-878b-fd0942de55ff-kube-api-access-2jgw9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.456135 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.456039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:14.770872 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:14.770843    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"]
Sep 18 22:56:14.772615 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:56:14.772586    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0390769c_f94f_4cbe_878b_fd0942de55ff.slice/crio-4a84edf5081a5caa8e3fff22cd7aab6edb0c5cbc697c4a85059039d087ce18c0 WatchSource:0}: Error finding container 4a84edf5081a5caa8e3fff22cd7aab6edb0c5cbc697c4a85059039d087ce18c0: Status 404 returned error can't find the container with id 4a84edf5081a5caa8e3fff22cd7aab6edb0c5cbc697c4a85059039d087ce18c0
Sep 18 22:56:15.004240 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:15.004201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" event={"ID":"0390769c-f94f-4cbe-878b-fd0942de55ff","Type":"ContainerStarted","Data":"8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737"}
Sep 18 22:56:15.004447 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:15.004266    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" event={"ID":"0390769c-f94f-4cbe-878b-fd0942de55ff","Type":"ContainerStarted","Data":"4a84edf5081a5caa8e3fff22cd7aab6edb0c5cbc697c4a85059039d087ce18c0"}
Sep 18 22:56:18.026201 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:18.026165    2582 generic.go:356] "Generic (PLEG): container finished" podID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerID="8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737" exitCode=0
Sep 18 22:56:18.026663 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:18.026250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" event={"ID":"0390769c-f94f-4cbe-878b-fd0942de55ff","Type":"ContainerDied","Data":"8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737"}
Sep 18 22:56:18.739775 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:18.739718    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:56:19.043457 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:19.043360    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" event={"ID":"0390769c-f94f-4cbe-878b-fd0942de55ff","Type":"ContainerStarted","Data":"6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e"}
Sep 18 22:56:19.073995 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:19.073919    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podStartSLOduration=2.135839931 podStartE2EDuration="5.07390262s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.938062689s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:56:19.07236356 +0000 UTC m=+3936.105542454" watchObservedRunningTime="2026-09-18 22:56:19.07390262 +0000 UTC m=+3936.107081572"
Sep 18 22:56:19.457148 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:19.457097    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:24.456983 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:24.456917    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:56:24.458656 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:24.458620    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:56:27.965683 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:27.965595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gj275"
Sep 18 22:56:28.740401 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:28.740350    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:56:34.457016 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:34.456970    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:56:38.740496 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:38.740449    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:56:44.457336 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:44.457282    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:56:48.740154 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:48.740097    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:56:54.456497 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:54.456436    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:56:58.740077 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:56:58.740026    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:57:04.456766 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:04.456705    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:57:08.740450 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:08.740384    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:57:14.456861 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:14.456807    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:57:18.739678 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:18.739625    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:57:24.457085 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:24.457035    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:57:28.740482 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:28.740419    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:57:34.467257 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:34.467223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:57:34.475046 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:34.475014    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:57:36.345303 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:36.345266    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"]
Sep 18 22:57:38.740419 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:38.740369    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Sep 18 22:57:48.755222 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:48.755183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:57:48.766881 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:48.766858    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:57:51.368575 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:51.368513    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" containerID="cri-o://6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e" gracePeriod=45
Sep 18 22:57:51.468405 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:51.468359    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 22:57:52.035498 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.035467    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:57:52.138415 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138387    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-home\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.138567 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138434    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-dshm\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.138567 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jgw9\" (UniqueName: \"kubernetes.io/projected/0390769c-f94f-4cbe-878b-fd0942de55ff-kube-api-access-2jgw9\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.138567 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-tmp-dir\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.138567 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0390769c-f94f-4cbe-878b-fd0942de55ff-tls-certs\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.138788 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138570    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-model-cache\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.138854 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138802    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:57:52.138854 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138826    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-kserve-provision-location\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.138984 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-home" (OuterVolumeSpecName: "home") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:57:52.138984 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.138890    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0390769c-f94f-4cbe-878b-fd0942de55ff-cabundle-cert\") pod \"0390769c-f94f-4cbe-878b-fd0942de55ff\" (UID: \"0390769c-f94f-4cbe-878b-fd0942de55ff\") "
Sep 18 22:57:52.139093 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.139060    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:57:52.139596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.139577    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.139853 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.139838    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.140002 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.139981    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.140136 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.140115    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0390769c-f94f-4cbe-878b-fd0942de55ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:57:52.141002 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.140974    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-dshm" (OuterVolumeSpecName: "dshm") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:57:52.141165 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.141144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0390769c-f94f-4cbe-878b-fd0942de55ff-kube-api-access-2jgw9" (OuterVolumeSpecName: "kube-api-access-2jgw9") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "kube-api-access-2jgw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:57:52.141165 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.141154    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0390769c-f94f-4cbe-878b-fd0942de55ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:57:52.155359 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.155334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0390769c-f94f-4cbe-878b-fd0942de55ff" (UID: "0390769c-f94f-4cbe-878b-fd0942de55ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:57:52.241255 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.241223    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jgw9\" (UniqueName: \"kubernetes.io/projected/0390769c-f94f-4cbe-878b-fd0942de55ff-kube-api-access-2jgw9\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.241255 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.241249    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0390769c-f94f-4cbe-878b-fd0942de55ff-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.241430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.241263    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.241430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.241276    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0390769c-f94f-4cbe-878b-fd0942de55ff-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.241430 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.241287    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0390769c-f94f-4cbe-878b-fd0942de55ff-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:57:52.649563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.649529    2582 generic.go:356] "Generic (PLEG): container finished" podID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerID="6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e" exitCode=0
Sep 18 22:57:52.649979 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.649578    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"
Sep 18 22:57:52.649979 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.649623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" event={"ID":"0390769c-f94f-4cbe-878b-fd0942de55ff","Type":"ContainerDied","Data":"6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e"}
Sep 18 22:57:52.649979 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.649681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh" event={"ID":"0390769c-f94f-4cbe-878b-fd0942de55ff","Type":"ContainerDied","Data":"4a84edf5081a5caa8e3fff22cd7aab6edb0c5cbc697c4a85059039d087ce18c0"}
Sep 18 22:57:52.649979 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.649708    2582 scope.go:117] "RemoveContainer" containerID="6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e"
Sep 18 22:57:52.659493 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.659476    2582 scope.go:117] "RemoveContainer" containerID="8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737"
Sep 18 22:57:52.668654 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.668629    2582 scope.go:117] "RemoveContainer" containerID="6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e"
Sep 18 22:57:52.668986 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:57:52.668964    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e\": container with ID starting with 6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e not found: ID does not exist" containerID="6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e"
Sep 18 22:57:52.669049 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.668995    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e"} err="failed to get container status \"6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e\": rpc error: code = NotFound desc = could not find container \"6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e\": container with ID starting with 6cdaaeb127bdc5a6172943f18f5d7ca53b8caee84bbad56d09597aef7aedd94e not found: ID does not exist"
Sep 18 22:57:52.669049 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.669020    2582 scope.go:117] "RemoveContainer" containerID="8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737"
Sep 18 22:57:52.669426 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:57:52.669380    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737\": container with ID starting with 8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737 not found: ID does not exist" containerID="8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737"
Sep 18 22:57:52.669426 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.669416    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737"} err="failed to get container status \"8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737\": rpc error: code = NotFound desc = could not find container \"8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737\": container with ID starting with 8dc51b8aeaf826ec2a0f712550e5217ce3c2b3e5eb2a10f0c5f4fbc3d5029737 not found: ID does not exist"
Sep 18 22:57:52.671707 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.671685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"]
Sep 18 22:57:52.675327 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.675306    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-559df56667-84zhh"]
Sep 18 22:57:52.975783 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:52.975702    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" path="/var/lib/kubelet/pods/0390769c-f94f-4cbe-878b-fd0942de55ff/volumes"
Sep 18 22:57:57.862536 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:57:57.862456    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"]
Sep 18 22:58:12.889304 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:12.889235    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main" containerID="cri-o://fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837" gracePeriod=45
Sep 18 22:58:13.507137 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.507080    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="llm-d-routing-sidecar" containerID="cri-o://731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8" gracePeriod=45
Sep 18 22:58:13.756995 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.756972    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:58:13.774353 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.774320    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerID="fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837" exitCode=0
Sep 18 22:58:13.774353 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.774346    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerID="731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8" exitCode=0
Sep 18 22:58:13.774563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.774375    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"
Sep 18 22:58:13.774563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.774387    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerDied","Data":"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"}
Sep 18 22:58:13.774563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.774444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerDied","Data":"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"}
Sep 18 22:58:13.774563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.774466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2" event={"ID":"0d174743-5feb-424d-aaa8-9896eda12d8c","Type":"ContainerDied","Data":"d3d34b3fda1a012db7f1c81cb137a9e8f0f91878fb323b8ea88f6ac65d57f998"}
Sep 18 22:58:13.774563 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.774479    2582 scope.go:117] "RemoveContainer" containerID="fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"
Sep 18 22:58:13.785587 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.785530    2582 scope.go:117] "RemoveContainer" containerID="abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"
Sep 18 22:58:13.795046 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.795026    2582 scope.go:117] "RemoveContainer" containerID="731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"
Sep 18 22:58:13.803050 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.803019    2582 scope.go:117] "RemoveContainer" containerID="fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"
Sep 18 22:58:13.803330 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:58:13.803310    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837\": container with ID starting with fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837 not found: ID does not exist" containerID="fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"
Sep 18 22:58:13.803400 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.803344    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"} err="failed to get container status \"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837\": rpc error: code = NotFound desc = could not find container \"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837\": container with ID starting with fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837 not found: ID does not exist"
Sep 18 22:58:13.803400 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.803373    2582 scope.go:117] "RemoveContainer" containerID="abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"
Sep 18 22:58:13.803718 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:58:13.803685    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a\": container with ID starting with abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a not found: ID does not exist" containerID="abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"
Sep 18 22:58:13.803797 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.803724    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"} err="failed to get container status \"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a\": rpc error: code = NotFound desc = could not find container \"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a\": container with ID starting with abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a not found: ID does not exist"
Sep 18 22:58:13.803797 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.803742    2582 scope.go:117] "RemoveContainer" containerID="731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"
Sep 18 22:58:13.804092 ip-10-0-142-209 kubenswrapper[2582]: E0918 22:58:13.804065    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8\": container with ID starting with 731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8 not found: ID does not exist" containerID="731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"
Sep 18 22:58:13.804159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.804097    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"} err="failed to get container status \"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8\": rpc error: code = NotFound desc = could not find container \"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8\": container with ID starting with 731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8 not found: ID does not exist"
Sep 18 22:58:13.804159 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.804111    2582 scope.go:117] "RemoveContainer" containerID="fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"
Sep 18 22:58:13.804401 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.804382    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837"} err="failed to get container status \"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837\": rpc error: code = NotFound desc = could not find container \"fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837\": container with ID starting with fb1f350ceeac359f44b754d2ca22474c3d8079b6947f5f96d29111b0867a5837 not found: ID does not exist"
Sep 18 22:58:13.804455 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.804403    2582 scope.go:117] "RemoveContainer" containerID="abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"
Sep 18 22:58:13.804659 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.804644    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a"} err="failed to get container status \"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a\": rpc error: code = NotFound desc = could not find container \"abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a\": container with ID starting with abe6cfd3b6dbe790c1683503093812d61934a35730545f738fded1279e56272a not found: ID does not exist"
Sep 18 22:58:13.804659 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.804659    2582 scope.go:117] "RemoveContainer" containerID="731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"
Sep 18 22:58:13.804887 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.804871    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8"} err="failed to get container status \"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8\": rpc error: code = NotFound desc = could not find container \"731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8\": container with ID starting with 731a67044d2b363d564f83e448a57d1111133abb1540df874f27cd640b80ced8 not found: ID does not exist"
Sep 18 22:58:13.829491 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-dshm\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.829593 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829504    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d174743-5feb-424d-aaa8-9896eda12d8c-cabundle-cert\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.829593 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-home\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.829593 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829566    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-model-cache\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.829770 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-kserve-provision-location\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.829770 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829639    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r4zh\" (UniqueName: \"kubernetes.io/projected/0d174743-5feb-424d-aaa8-9896eda12d8c-kube-api-access-5r4zh\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.829770 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-tmp-dir\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.829770 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d174743-5feb-424d-aaa8-9896eda12d8c-tls-certs\") pod \"0d174743-5feb-424d-aaa8-9896eda12d8c\" (UID: \"0d174743-5feb-424d-aaa8-9896eda12d8c\") "
Sep 18 22:58:13.830043 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829896    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.830043 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.829913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d174743-5feb-424d-aaa8-9896eda12d8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:58:13.830043 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.830014    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d174743-5feb-424d-aaa8-9896eda12d8c-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.830043 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.830035    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.830443 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.830418    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-home" (OuterVolumeSpecName: "home") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.831880 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.831851    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-dshm" (OuterVolumeSpecName: "dshm") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.832139 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.832109    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d174743-5feb-424d-aaa8-9896eda12d8c-kube-api-access-5r4zh" (OuterVolumeSpecName: "kube-api-access-5r4zh") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "kube-api-access-5r4zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:58:13.832385 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.832364    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d174743-5feb-424d-aaa8-9896eda12d8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:58:13.835566 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.835543    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.844513 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.844487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d174743-5feb-424d-aaa8-9896eda12d8c" (UID: "0d174743-5feb-424d-aaa8-9896eda12d8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:58:13.930596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.930528    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.930596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.930553    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.930596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.930563    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r4zh\" (UniqueName: \"kubernetes.io/projected/0d174743-5feb-424d-aaa8-9896eda12d8c-kube-api-access-5r4zh\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.930596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.930572    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.930596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.930580    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d174743-5feb-424d-aaa8-9896eda12d8c-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:13.930596 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:13.930587    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d174743-5feb-424d-aaa8-9896eda12d8c-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 22:58:14.098783 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:14.098751    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"]
Sep 18 22:58:14.102335 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:14.102308    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-5fb5c479f5-smfn2"]
Sep 18 22:58:14.974529 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:14.974487    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" path="/var/lib/kubelet/pods/0d174743-5feb-424d-aaa8-9896eda12d8c/volumes"
Sep 18 22:58:33.924721 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.924686    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"]
Sep 18 22:58:33.925206 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925129    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="storage-initializer"
Sep 18 22:58:33.925206 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925149    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="storage-initializer"
Sep 18 22:58:33.925206 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925162    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main"
Sep 18 22:58:33.925206 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925171    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main"
Sep 18 22:58:33.925206 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925190    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main"
Sep 18 22:58:33.925206 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925199    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main"
Sep 18 22:58:33.925404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925218    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="storage-initializer"
Sep 18 22:58:33.925404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925228    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="storage-initializer"
Sep 18 22:58:33.925404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925248    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="llm-d-routing-sidecar"
Sep 18 22:58:33.925404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925258    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="llm-d-routing-sidecar"
Sep 18 22:58:33.925404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925376    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="llm-d-routing-sidecar"
Sep 18 22:58:33.925404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925386    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d174743-5feb-424d-aaa8-9896eda12d8c" containerName="main"
Sep 18 22:58:33.925404 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.925397    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0390769c-f94f-4cbe-878b-fd0942de55ff" containerName="main"
Sep 18 22:58:33.928624 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.928604    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.930865 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.930823    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 22:58:33.930865 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.930845    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 22:58:33.931085 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.930990    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-fp2ct\""
Sep 18 22:58:33.931085 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.931023    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 22:58:33.931223 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.931104    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 22:58:33.931223 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.931133    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 22:58:33.939044 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.939022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"]
Sep 18 22:58:33.989342 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-home\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.989484 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-model-cache\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.989484 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-tmp-dir\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.989484 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-dshm\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.989484 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8jj\" (UniqueName: \"kubernetes.io/projected/af270e2e-cc38-4388-b3c3-54570c30fa68-kube-api-access-2v8jj\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.989641 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af270e2e-cc38-4388-b3c3-54570c30fa68-tls-certs\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.989641 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-kserve-provision-location\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:33.989641 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:33.989624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af270e2e-cc38-4388-b3c3-54570c30fa68-cabundle-cert\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090162 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-home\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-model-cache\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-tmp-dir\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-dshm\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8jj\" (UniqueName: \"kubernetes.io/projected/af270e2e-cc38-4388-b3c3-54570c30fa68-kube-api-access-2v8jj\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af270e2e-cc38-4388-b3c3-54570c30fa68-tls-certs\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090338 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-kserve-provision-location\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090667 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af270e2e-cc38-4388-b3c3-54570c30fa68-cabundle-cert\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090667 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-model-cache\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090667 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-tmp-dir\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090858 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090836    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-home\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.090909 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.090860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-kserve-provision-location\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.091239 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.091214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af270e2e-cc38-4388-b3c3-54570c30fa68-cabundle-cert\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.092611 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.092593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-dshm\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.092911 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.092890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af270e2e-cc38-4388-b3c3-54570c30fa68-tls-certs\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.098640 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.098615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8jj\" (UniqueName: \"kubernetes.io/projected/af270e2e-cc38-4388-b3c3-54570c30fa68-kube-api-access-2v8jj\") pod \"stop-feature-test-kserve-74b655bbc5-9qx6h\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.240729 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.240625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:34.520892 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.520866    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"]
Sep 18 22:58:34.523285 ip-10-0-142-209 kubenswrapper[2582]: W0918 22:58:34.523251    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf270e2e_cc38_4388_b3c3_54570c30fa68.slice/crio-2f8217ddf3f1f2865ab156877f8f410373819fe4c34a36cfa1c2cc6315a315e8 WatchSource:0}: Error finding container 2f8217ddf3f1f2865ab156877f8f410373819fe4c34a36cfa1c2cc6315a315e8: Status 404 returned error can't find the container with id 2f8217ddf3f1f2865ab156877f8f410373819fe4c34a36cfa1c2cc6315a315e8
Sep 18 22:58:34.891227 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.891189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" event={"ID":"af270e2e-cc38-4388-b3c3-54570c30fa68","Type":"ContainerStarted","Data":"19db69fb7d84a5a89755577d0fe7ee8121b45e7ee8109a651c4eacd32f964799"}
Sep 18 22:58:34.891392 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:34.891254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" event={"ID":"af270e2e-cc38-4388-b3c3-54570c30fa68","Type":"ContainerStarted","Data":"2f8217ddf3f1f2865ab156877f8f410373819fe4c34a36cfa1c2cc6315a315e8"}
Sep 18 22:58:36.903844 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:36.903809    2582 generic.go:356] "Generic (PLEG): container finished" podID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerID="19db69fb7d84a5a89755577d0fe7ee8121b45e7ee8109a651c4eacd32f964799" exitCode=0
Sep 18 22:58:36.904203 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:36.903880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" event={"ID":"af270e2e-cc38-4388-b3c3-54570c30fa68","Type":"ContainerDied","Data":"19db69fb7d84a5a89755577d0fe7ee8121b45e7ee8109a651c4eacd32f964799"}
Sep 18 22:58:37.913704 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:37.913665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" event={"ID":"af270e2e-cc38-4388-b3c3-54570c30fa68","Type":"ContainerStarted","Data":"72256daf76e53044ed566883cc6b42b5ffb6553495cd2c4403d93ca002b13c17"}
Sep 18 22:58:37.939982 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:37.939905    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podStartSLOduration=2.73995511 podStartE2EDuration="4.939882355s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.199927245s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:58:37.936488221 +0000 UTC m=+4074.969667115" watchObservedRunningTime="2026-09-18 22:58:37.939882355 +0000 UTC m=+4074.973061251"
Sep 18 22:58:38.240764 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:38.240688    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:44.241574 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:44.241531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 22:58:44.243312 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:44.243276    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 22:58:54.241982 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:58:54.241904    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 22:59:04.241743 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:59:04.241687    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 22:59:14.241704 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:59:14.241656    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 22:59:24.241283 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:59:24.241233    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 22:59:34.241358 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:59:34.241279    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 22:59:44.242093 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:59:44.242036    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 22:59:54.241875 ip-10-0-142-209 kubenswrapper[2582]: I0918 22:59:54.241825    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 23:00:04.242034 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:04.241979    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 23:00:14.241266 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:14.241218    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 23:00:24.241994 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:24.241914    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 23:00:34.251649 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:34.251614    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 23:00:34.259742 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:34.259722    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 23:00:35.937929 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:35.937896    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"]
Sep 18 23:00:45.387508 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:45.387483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:00:45.393632 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:45.393608    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:00:45.402107 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:45.402085    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:00:45.407259 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:45.407239    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:00:50.961350 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:50.961302    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" containerID="cri-o://72256daf76e53044ed566883cc6b42b5ffb6553495cd2c4403d93ca002b13c17" gracePeriod=45
Sep 18 23:00:51.253285 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.253190    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 23:00:51.680182 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.680145    2582 generic.go:356] "Generic (PLEG): container finished" podID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerID="72256daf76e53044ed566883cc6b42b5ffb6553495cd2c4403d93ca002b13c17" exitCode=0
Sep 18 23:00:51.680353 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.680181    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" event={"ID":"af270e2e-cc38-4388-b3c3-54570c30fa68","Type":"ContainerDied","Data":"72256daf76e53044ed566883cc6b42b5ffb6553495cd2c4403d93ca002b13c17"}
Sep 18 23:00:51.710821 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.710796    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 23:00:51.744164 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744135    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v8jj\" (UniqueName: \"kubernetes.io/projected/af270e2e-cc38-4388-b3c3-54570c30fa68-kube-api-access-2v8jj\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744303 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744200    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-dshm\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744303 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744235    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-tmp-dir\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744303 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744263    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-home\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744303 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744287    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af270e2e-cc38-4388-b3c3-54570c30fa68-cabundle-cert\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744544 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744320    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af270e2e-cc38-4388-b3c3-54570c30fa68-tls-certs\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744544 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-kserve-provision-location\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744544 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-model-cache\") pod \"af270e2e-cc38-4388-b3c3-54570c30fa68\" (UID: \"af270e2e-cc38-4388-b3c3-54570c30fa68\") "
Sep 18 23:00:51.744796 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744763    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af270e2e-cc38-4388-b3c3-54570c30fa68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:00:51.745093 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.744908    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-model-cache" (OuterVolumeSpecName: "model-cache") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:00:51.745229 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.745204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-home" (OuterVolumeSpecName: "home") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:00:51.746628 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.746604    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-dshm" (OuterVolumeSpecName: "dshm") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:00:51.746753 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.746730    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af270e2e-cc38-4388-b3c3-54570c30fa68-kube-api-access-2v8jj" (OuterVolumeSpecName: "kube-api-access-2v8jj") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "kube-api-access-2v8jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:00:51.747243 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.747218    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af270e2e-cc38-4388-b3c3-54570c30fa68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:00:51.758370 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.758345    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:00:51.781631 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.781603    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af270e2e-cc38-4388-b3c3-54570c30fa68" (UID: "af270e2e-cc38-4388-b3c3-54570c30fa68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:00:51.845821 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845760    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:51.845821 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845782    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:51.845821 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845792    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:51.845821 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845803    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af270e2e-cc38-4388-b3c3-54570c30fa68-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:51.845821 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845811    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af270e2e-cc38-4388-b3c3-54570c30fa68-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:51.845821 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845819    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:51.846118 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845828    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af270e2e-cc38-4388-b3c3-54570c30fa68-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:51.846118 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:51.845838    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v8jj\" (UniqueName: \"kubernetes.io/projected/af270e2e-cc38-4388-b3c3-54570c30fa68-kube-api-access-2v8jj\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:00:52.687924 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:52.687888    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"
Sep 18 23:00:52.687924 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:52.687915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h" event={"ID":"af270e2e-cc38-4388-b3c3-54570c30fa68","Type":"ContainerDied","Data":"2f8217ddf3f1f2865ab156877f8f410373819fe4c34a36cfa1c2cc6315a315e8"}
Sep 18 23:00:52.688445 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:52.688010    2582 scope.go:117] "RemoveContainer" containerID="72256daf76e53044ed566883cc6b42b5ffb6553495cd2c4403d93ca002b13c17"
Sep 18 23:00:52.698178 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:52.698162    2582 scope.go:117] "RemoveContainer" containerID="19db69fb7d84a5a89755577d0fe7ee8121b45e7ee8109a651c4eacd32f964799"
Sep 18 23:00:52.711083 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:52.711062    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"]
Sep 18 23:00:52.714742 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:52.714721    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-9qx6h"]
Sep 18 23:00:52.974979 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:52.974877    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" path="/var/lib/kubelet/pods/af270e2e-cc38-4388-b3c3-54570c30fa68/volumes"
Sep 18 23:00:56.075464 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.075432    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"]
Sep 18 23:00:56.075848 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.075814    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main"
Sep 18 23:00:56.075848 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.075826    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main"
Sep 18 23:00:56.075848 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.075838    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="storage-initializer"
Sep 18 23:00:56.075848 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.075844    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="storage-initializer"
Sep 18 23:00:56.076020 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.075901    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="af270e2e-cc38-4388-b3c3-54570c30fa68" containerName="main"
Sep 18 23:00:56.079167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.079148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.081880 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.081843    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-fp2ct\""
Sep 18 23:00:56.082074 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.081903    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:00:56.082074 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.081914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 23:00:56.082074 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.081845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 23:00:56.082074 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.081914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 23:00:56.082074 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.081845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 23:00:56.094859 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.094835    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"]
Sep 18 23:00:56.183311 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-dshm\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.183311 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg5qv\" (UniqueName: \"kubernetes.io/projected/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kube-api-access-bg5qv\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.183534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-model-cache\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.183534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tmp-dir\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.183534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tls-certs\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.183534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-cabundle-cert\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.183534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kserve-provision-location\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.183728 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.183550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-home\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.284811 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.284777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-dshm\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.284811 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.284815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg5qv\" (UniqueName: \"kubernetes.io/projected/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kube-api-access-bg5qv\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285089 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.284846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-model-cache\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285089 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.284889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tmp-dir\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285089 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.284914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tls-certs\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285089 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.284935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-cabundle-cert\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285089 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.285041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kserve-provision-location\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285372 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.285114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-home\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285372 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.285353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tmp-dir\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285486 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.285370    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-model-cache\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285486 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.285458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kserve-provision-location\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285600 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.285577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-home\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.285712 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.285696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-cabundle-cert\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.287186 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.287167    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-dshm\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.287574 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.287555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tls-certs\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.292866 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.292845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg5qv\" (UniqueName: \"kubernetes.io/projected/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kube-api-access-bg5qv\") pod \"stop-feature-test-kserve-74b655bbc5-fhht6\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.389826 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.389798    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:00:56.670429 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.670401    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"]
Sep 18 23:00:56.672318 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:00:56.672287    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea4fbb2_468f_4ffc_9c47_982b0cdf8b80.slice/crio-7b46f888e3223ee77461814582cf5232e95f0642237d17692d595804216017a5 WatchSource:0}: Error finding container 7b46f888e3223ee77461814582cf5232e95f0642237d17692d595804216017a5: Status 404 returned error can't find the container with id 7b46f888e3223ee77461814582cf5232e95f0642237d17692d595804216017a5
Sep 18 23:00:56.709710 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:56.709672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" event={"ID":"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80","Type":"ContainerStarted","Data":"7b46f888e3223ee77461814582cf5232e95f0642237d17692d595804216017a5"}
Sep 18 23:00:57.720505 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:57.720448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" event={"ID":"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80","Type":"ContainerStarted","Data":"eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a"}
Sep 18 23:00:59.733707 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:59.733670    2582 generic.go:356] "Generic (PLEG): container finished" podID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerID="eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a" exitCode=0
Sep 18 23:00:59.734142 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:00:59.733741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" event={"ID":"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80","Type":"ContainerDied","Data":"eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a"}
Sep 18 23:01:00.741320 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:00.741282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" event={"ID":"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80","Type":"ContainerStarted","Data":"1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa"}
Sep 18 23:01:00.776338 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:00.776269    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podStartSLOduration=2.593501487 podStartE2EDuration="4.776254796s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.182753309s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:01:00.775042121 +0000 UTC m=+4217.808221015" watchObservedRunningTime="2026-09-18 23:01:00.776254796 +0000 UTC m=+4217.809433715"
Sep 18 23:01:01.390925 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:01.390888    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:01:06.390252 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:06.390214    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:01:06.392053 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:06.392023    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:01:16.390329 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:16.390288    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:01:26.390941 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:26.390899    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:01:36.390438 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:36.390396    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:01:45.164348 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.164313    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 23:01:45.168220 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.168197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.171182 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.171158    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:01:45.171316 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.171294    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-bjm8j\""
Sep 18 23:01:45.172270 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.172244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 23:01:45.172399 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.172271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 23:01:45.172399 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.172334    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-82vx4\""
Sep 18 23:01:45.172592 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.172572    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 23:01:45.172657 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.172574    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 23:01:45.181146 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.181124    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 23:01:45.230510 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.230675 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.230675 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230559    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71feaaca-e27c-4a5a-89e6-1086452684f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.230675 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.230675 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.230814 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.230814 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtglq\" (UniqueName: \"kubernetes.io/projected/71feaaca-e27c-4a5a-89e6-1086452684f2-kube-api-access-wtglq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.230885 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.230835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71feaaca-e27c-4a5a-89e6-1086452684f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332041 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.331997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71feaaca-e27c-4a5a-89e6-1086452684f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332236 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332236 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332236 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71feaaca-e27c-4a5a-89e6-1086452684f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332236 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332236 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332504 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332504 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtglq\" (UniqueName: \"kubernetes.io/projected/71feaaca-e27c-4a5a-89e6-1086452684f2-kube-api-access-wtglq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332504 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332689 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332751 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332834 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71feaaca-e27c-4a5a-89e6-1086452684f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.332922 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.332903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.334447 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.334428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.334602 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.334583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71feaaca-e27c-4a5a-89e6-1086452684f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.342845 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.342822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtglq\" (UniqueName: \"kubernetes.io/projected/71feaaca-e27c-4a5a-89e6-1086452684f2-kube-api-access-wtglq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.482088 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.482009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:01:45.774787 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:45.774711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 23:01:45.778241 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:01:45.778212    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71feaaca_e27c_4a5a_89e6_1086452684f2.slice/crio-870d704ecf054b6b047d5eee4a77a493444336f292e7ec704d6568b564bd7f71 WatchSource:0}: Error finding container 870d704ecf054b6b047d5eee4a77a493444336f292e7ec704d6568b564bd7f71: Status 404 returned error can't find the container with id 870d704ecf054b6b047d5eee4a77a493444336f292e7ec704d6568b564bd7f71
Sep 18 23:01:46.006835 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:46.006794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"71feaaca-e27c-4a5a-89e6-1086452684f2","Type":"ContainerStarted","Data":"81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2"}
Sep 18 23:01:46.007021 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:46.006867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"71feaaca-e27c-4a5a-89e6-1086452684f2","Type":"ContainerStarted","Data":"870d704ecf054b6b047d5eee4a77a493444336f292e7ec704d6568b564bd7f71"}
Sep 18 23:01:46.390774 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:46.390726    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:01:49.037201 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:49.037163    2582 generic.go:356] "Generic (PLEG): container finished" podID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerID="81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2" exitCode=0
Sep 18 23:01:49.037694 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:49.037205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"71feaaca-e27c-4a5a-89e6-1086452684f2","Type":"ContainerDied","Data":"81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2"}
Sep 18 23:01:50.054594 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:50.052795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"71feaaca-e27c-4a5a-89e6-1086452684f2","Type":"ContainerStarted","Data":"561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891"}
Sep 18 23:01:50.087803 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:50.087734    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.6787788 podStartE2EDuration="5.087713489s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.408934689s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:01:50.084320348 +0000 UTC m=+4267.117499253" watchObservedRunningTime="2026-09-18 23:01:50.087713489 +0000 UTC m=+4267.120892382"
Sep 18 23:01:56.390524 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:01:56.390481    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:02:06.390395 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:02:06.390346    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:02:16.391043 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:02:16.390996    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:02:26.390448 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:02:26.390398    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:02:36.390805 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:02:36.390708    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:02:46.390466 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:02:46.390425    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:02:56.391206 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:02:56.391162    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:03:06.400820 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:06.400790    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:03:06.409194 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:06.409169    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:03:07.942371 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:07.942331    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"]
Sep 18 23:03:22.969011 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:22.968919    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" containerID="cri-o://1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa" gracePeriod=45
Sep 18 23:03:23.401965 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:23.401906    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 23:03:23.917329 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:23.917295    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:03:24.040560 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040478    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-home\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.040560 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040537    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-model-cache\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.041106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kserve-provision-location\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.041106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040600    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-cabundle-cert\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.041106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040646    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tls-certs\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.041106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040720    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg5qv\" (UniqueName: \"kubernetes.io/projected/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kube-api-access-bg5qv\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.041106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040757    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tmp-dir\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.041106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-dshm\") pod \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\" (UID: \"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80\") "
Sep 18 23:03:24.041106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.040874    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:24.041493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.041146    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.041493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.041282    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-home" (OuterVolumeSpecName: "home") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:24.041493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.041372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:03:24.043584 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.043562    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-dshm" (OuterVolumeSpecName: "dshm") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:24.044233 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.044197    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:03:24.044349 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.044242    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kube-api-access-bg5qv" (OuterVolumeSpecName: "kube-api-access-bg5qv") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "kube-api-access-bg5qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:03:24.049368 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.049345    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:24.058733 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.058701    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" (UID: "1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:03:24.141778 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.141741    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.141778 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.141773    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg5qv\" (UniqueName: \"kubernetes.io/projected/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kube-api-access-bg5qv\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.141778 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.141783    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.142029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.141793    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.142029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.141800    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.142029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.141808    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.142029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.141816    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:03:24.636655 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.636618    2582 generic.go:356] "Generic (PLEG): container finished" podID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerID="1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa" exitCode=0
Sep 18 23:03:24.636860 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.636678    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"
Sep 18 23:03:24.636860 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.636700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" event={"ID":"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80","Type":"ContainerDied","Data":"1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa"}
Sep 18 23:03:24.636860 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.636781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6" event={"ID":"1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80","Type":"ContainerDied","Data":"7b46f888e3223ee77461814582cf5232e95f0642237d17692d595804216017a5"}
Sep 18 23:03:24.636860 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.636803    2582 scope.go:117] "RemoveContainer" containerID="1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa"
Sep 18 23:03:24.648620 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.648604    2582 scope.go:117] "RemoveContainer" containerID="eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a"
Sep 18 23:03:24.657048 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.657031    2582 scope.go:117] "RemoveContainer" containerID="1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa"
Sep 18 23:03:24.657310 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:03:24.657292    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa\": container with ID starting with 1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa not found: ID does not exist" containerID="1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa"
Sep 18 23:03:24.657360 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.657319    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa"} err="failed to get container status \"1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa\": rpc error: code = NotFound desc = could not find container \"1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa\": container with ID starting with 1e40df52e19fe16a9c6fa85ff0f4dea702c3d45d5ac12d4b284faf8afa52cdfa not found: ID does not exist"
Sep 18 23:03:24.657360 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.657336    2582 scope.go:117] "RemoveContainer" containerID="eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a"
Sep 18 23:03:24.657646 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:03:24.657628    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a\": container with ID starting with eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a not found: ID does not exist" containerID="eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a"
Sep 18 23:03:24.657680 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.657654    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a"} err="failed to get container status \"eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a\": rpc error: code = NotFound desc = could not find container \"eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a\": container with ID starting with eb62a4b6b21d88b8d6d97fbeb8c7b385a247ea8262921fc963779dfe8439807a not found: ID does not exist"
Sep 18 23:03:24.663167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.663144    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"]
Sep 18 23:03:24.667010 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.666989    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-74b655bbc5-fhht6"]
Sep 18 23:03:24.975398 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:24.975305    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" path="/var/lib/kubelet/pods/1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80/volumes"
Sep 18 23:03:31.636870 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:31.636815    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-209.ec2.internal" podUID="6e9479c83aec32841bba93774c695096" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 23:03:46.803663 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:03:46.803626    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 23:04:01.827277 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:01.827191    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerName="main" containerID="cri-o://561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891" gracePeriod=45
Sep 18 23:04:02.579666 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.579636    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:04:02.677107 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677075    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-home\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677294 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677133    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-kserve-provision-location\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677294 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677156    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71feaaca-e27c-4a5a-89e6-1086452684f2-tls-certs\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677294 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677177    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71feaaca-e27c-4a5a-89e6-1086452684f2-cabundle-cert\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677294 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677199    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-model-cache\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677294 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677234    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtglq\" (UniqueName: \"kubernetes.io/projected/71feaaca-e27c-4a5a-89e6-1086452684f2-kube-api-access-wtglq\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677294 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677265    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-tmp-dir\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677294 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677294    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-dshm\") pod \"71feaaca-e27c-4a5a-89e6-1086452684f2\" (UID: \"71feaaca-e27c-4a5a-89e6-1086452684f2\") "
Sep 18 23:04:02.677672 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677567    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:04:02.677831 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677798    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71feaaca-e27c-4a5a-89e6-1086452684f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:04:02.677998 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.677913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-home" (OuterVolumeSpecName: "home") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:04:02.679767 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.679739    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71feaaca-e27c-4a5a-89e6-1086452684f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:04:02.679884 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.679765    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71feaaca-e27c-4a5a-89e6-1086452684f2-kube-api-access-wtglq" (OuterVolumeSpecName: "kube-api-access-wtglq") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "kube-api-access-wtglq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:04:02.679884 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.679828    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-dshm" (OuterVolumeSpecName: "dshm") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:04:02.688309 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.688284    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:04:02.691393 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.691366    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71feaaca-e27c-4a5a-89e6-1086452684f2" (UID: "71feaaca-e27c-4a5a-89e6-1086452684f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:04:02.778206 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778155    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.778206 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778197    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71feaaca-e27c-4a5a-89e6-1086452684f2-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.778206 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778211    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71feaaca-e27c-4a5a-89e6-1086452684f2-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.778206 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778222    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.778493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778236    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtglq\" (UniqueName: \"kubernetes.io/projected/71feaaca-e27c-4a5a-89e6-1086452684f2-kube-api-access-wtglq\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.778493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778268    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.778493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778281    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.778493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.778291    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71feaaca-e27c-4a5a-89e6-1086452684f2-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:04:02.846987 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.846924    2582 generic.go:356] "Generic (PLEG): container finished" podID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerID="561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891" exitCode=0
Sep 18 23:04:02.847457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.846994    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 23:04:02.847457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.847015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"71feaaca-e27c-4a5a-89e6-1086452684f2","Type":"ContainerDied","Data":"561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891"}
Sep 18 23:04:02.847457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.847122    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"71feaaca-e27c-4a5a-89e6-1086452684f2","Type":"ContainerDied","Data":"870d704ecf054b6b047d5eee4a77a493444336f292e7ec704d6568b564bd7f71"}
Sep 18 23:04:02.847457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.847159    2582 scope.go:117] "RemoveContainer" containerID="561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891"
Sep 18 23:04:02.857002 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.856981    2582 scope.go:117] "RemoveContainer" containerID="81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2"
Sep 18 23:04:02.865809 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.865793    2582 scope.go:117] "RemoveContainer" containerID="561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891"
Sep 18 23:04:02.866079 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:04:02.866061    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891\": container with ID starting with 561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891 not found: ID does not exist" containerID="561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891"
Sep 18 23:04:02.866135 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.866087    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891"} err="failed to get container status \"561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891\": rpc error: code = NotFound desc = could not find container \"561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891\": container with ID starting with 561145a9d50da52dd1f650fa4c7f545c431e5b8c93b6609932e4f4a1adacf891 not found: ID does not exist"
Sep 18 23:04:02.866135 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.866102    2582 scope.go:117] "RemoveContainer" containerID="81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2"
Sep 18 23:04:02.866333 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:04:02.866318    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2\": container with ID starting with 81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2 not found: ID does not exist" containerID="81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2"
Sep 18 23:04:02.866380 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.866338    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2"} err="failed to get container status \"81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2\": rpc error: code = NotFound desc = could not find container \"81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2\": container with ID starting with 81967f3f3b3ec9cec90fdbd4954f5b41c3ba088d605b62e38bf134fb1ba774c2 not found: ID does not exist"
Sep 18 23:04:02.870685 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.870663    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 23:04:02.874769 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.874748    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 23:04:02.974187 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:02.974106    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71feaaca-e27c-4a5a-89e6-1086452684f2" path="/var/lib/kubelet/pods/71feaaca-e27c-4a5a-89e6-1086452684f2/volumes"
Sep 18 23:04:17.820231 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820187    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"]
Sep 18 23:04:17.820621 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820566    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerName="storage-initializer"
Sep 18 23:04:17.820621 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820578    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerName="storage-initializer"
Sep 18 23:04:17.820621 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820594    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="storage-initializer"
Sep 18 23:04:17.820621 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820599    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="storage-initializer"
Sep 18 23:04:17.820621 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820615    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main"
Sep 18 23:04:17.820621 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820622    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main"
Sep 18 23:04:17.820824 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820635    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerName="main"
Sep 18 23:04:17.820824 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820642    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerName="main"
Sep 18 23:04:17.820824 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820703    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="71feaaca-e27c-4a5a-89e6-1086452684f2" containerName="main"
Sep 18 23:04:17.820824 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.820715    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ea4fbb2-468f-4ffc-9c47-982b0cdf8b80" containerName="main"
Sep 18 23:04:17.825640 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.825623    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.828227 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.828198    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:04:17.828377 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.828253    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 18 23:04:17.828457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.828275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-rmgkm\""
Sep 18 23:04:17.828457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.828396    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 18 23:04:17.828457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.828293    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 18 23:04:17.828888 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.828861    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 18 23:04:17.838577 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.838553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"]
Sep 18 23:04:17.908956 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.908912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-dshm\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.909124 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.908965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-model-cache\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.909124 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.909020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tls-certs\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.909124 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.909039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-home\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.909124 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.909056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tmp-dir\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.909124 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.909074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.909124 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.909092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsncc\" (UniqueName: \"kubernetes.io/projected/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kube-api-access-jsncc\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:17.909405 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:17.909149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a169-f35b-4ff0-b79f-bf54cd607f92-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.009905 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.009865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-model-cache\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010108 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.009914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tls-certs\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010108 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.009958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-home\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010108 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.009983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tmp-dir\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010108 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010108 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsncc\" (UniqueName: \"kubernetes.io/projected/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kube-api-access-jsncc\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010108 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a169-f35b-4ff0-b79f-bf54cd607f92-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010443 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-dshm\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010512 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-home\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010512 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-model-cache\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010602 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tmp-dir\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010688 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.010918 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.010898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a169-f35b-4ff0-b79f-bf54cd607f92-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.012429 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.012407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-dshm\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.012613 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.012596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tls-certs\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.017457 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.017434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsncc\" (UniqueName: \"kubernetes.io/projected/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kube-api-access-jsncc\") pod \"lora-single-adapter-test-kserve-7bbf7dc599-pkcc7\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.136285 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.136255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:18.423460 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.423431    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"]
Sep 18 23:04:18.425131 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:04:18.425100    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b81a169_f35b_4ff0_b79f_bf54cd607f92.slice/crio-4841f122a1d290e74557dd5402df1f874927cf2fdca016850f72178bb8ac1fbe WatchSource:0}: Error finding container 4841f122a1d290e74557dd5402df1f874927cf2fdca016850f72178bb8ac1fbe: Status 404 returned error can't find the container with id 4841f122a1d290e74557dd5402df1f874927cf2fdca016850f72178bb8ac1fbe
Sep 18 23:04:18.935834 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.935797    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" event={"ID":"0b81a169-f35b-4ff0-b79f-bf54cd607f92","Type":"ContainerStarted","Data":"3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d"}
Sep 18 23:04:18.936248 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:18.935857    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" event={"ID":"0b81a169-f35b-4ff0-b79f-bf54cd607f92","Type":"ContainerStarted","Data":"4841f122a1d290e74557dd5402df1f874927cf2fdca016850f72178bb8ac1fbe"}
Sep 18 23:04:21.956291 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:21.956252    2582 generic.go:356] "Generic (PLEG): container finished" podID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerID="3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d" exitCode=0
Sep 18 23:04:21.956903 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:21.956319    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" event={"ID":"0b81a169-f35b-4ff0-b79f-bf54cd607f92","Type":"ContainerDied","Data":"3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d"}
Sep 18 23:04:22.964851 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:22.964809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" event={"ID":"0b81a169-f35b-4ff0-b79f-bf54cd607f92","Type":"ContainerStarted","Data":"4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b"}
Sep 18 23:04:22.990355 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:22.990305    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podStartSLOduration=3.289407422 podStartE2EDuration="5.990287698s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.700880276s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:04:22.987119184 +0000 UTC m=+4420.020298114" watchObservedRunningTime="2026-09-18 23:04:22.990287698 +0000 UTC m=+4420.023466591"
Sep 18 23:04:23.137063 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:23.137026    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:28.136938 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:28.136896    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:04:28.138413 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:28.138382    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:04:38.137027 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:38.136974    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:04:48.137666 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:48.137617    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:04:58.137137 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:04:58.137089    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:05:08.136813 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:08.136757    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:05:18.137213 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:18.137167    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:05:28.137255 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:28.137161    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:05:38.146568 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:38.146537    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:05:38.154721 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:38.154696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:05:45.432472 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:45.432443    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:05:45.438936 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:45.438897    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:05:45.448018 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:45.447998    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:05:45.453726 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:05:45.453705    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:06:03.695372 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:03.695286    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7bbf7dc599-pkcc7_0b81a169-f35b-4ff0-b79f-bf54cd607f92/storage-initializer/0.log"
Sep 18 23:06:03.706049 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:03.706017    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7bbf7dc599-pkcc7_0b81a169-f35b-4ff0-b79f-bf54cd607f92/main/0.log"
Sep 18 23:06:04.711915 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:04.711875    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"]
Sep 18 23:06:19.735181 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:19.735115    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" containerID="cri-o://4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b" gracePeriod=45
Sep 18 23:06:20.147172 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.147124    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 18 23:06:20.386838 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.386811    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:06:20.490693 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490606    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-model-cache\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.490693 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kserve-provision-location\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.490693 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a169-f35b-4ff0-b79f-bf54cd607f92-cabundle-cert\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.490693 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tmp-dir\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.491072 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490712    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsncc\" (UniqueName: \"kubernetes.io/projected/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kube-api-access-jsncc\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.491072 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490747    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-home\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.491072 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tls-certs\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.491072 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490791    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-dshm\") pod \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\" (UID: \"0b81a169-f35b-4ff0-b79f-bf54cd607f92\") "
Sep 18 23:06:20.491072 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.490936    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:20.491337 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.491090    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b81a169-f35b-4ff0-b79f-bf54cd607f92-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:06:20.491337 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.491228    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.491337 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.491251    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a169-f35b-4ff0-b79f-bf54cd607f92-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.491495 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.491442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-home" (OuterVolumeSpecName: "home") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:20.493314 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.493291    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-dshm" (OuterVolumeSpecName: "dshm") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:20.493429 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.493353    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kube-api-access-jsncc" (OuterVolumeSpecName: "kube-api-access-jsncc") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "kube-api-access-jsncc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:06:20.493807 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.493783    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:06:20.504528 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.504504    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:20.516102 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.516079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b81a169-f35b-4ff0-b79f-bf54cd607f92" (UID: "0b81a169-f35b-4ff0-b79f-bf54cd607f92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:06:20.591843 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.591821    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.591843 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.591843    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsncc\" (UniqueName: \"kubernetes.io/projected/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kube-api-access-jsncc\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.592015 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.591852    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.592015 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.591862    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b81a169-f35b-4ff0-b79f-bf54cd607f92-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.592015 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.591869    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.592015 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.591877    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b81a169-f35b-4ff0-b79f-bf54cd607f92-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:06:20.646291 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.646258    2582 generic.go:356] "Generic (PLEG): container finished" podID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerID="4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b" exitCode=0
Sep 18 23:06:20.646450 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.646320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"
Sep 18 23:06:20.646450 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.646324    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" event={"ID":"0b81a169-f35b-4ff0-b79f-bf54cd607f92","Type":"ContainerDied","Data":"4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b"}
Sep 18 23:06:20.646450 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.646424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7" event={"ID":"0b81a169-f35b-4ff0-b79f-bf54cd607f92","Type":"ContainerDied","Data":"4841f122a1d290e74557dd5402df1f874927cf2fdca016850f72178bb8ac1fbe"}
Sep 18 23:06:20.646628 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.646455    2582 scope.go:117] "RemoveContainer" containerID="4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b"
Sep 18 23:06:20.655977 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.655958    2582 scope.go:117] "RemoveContainer" containerID="3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d"
Sep 18 23:06:20.664309 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.664293    2582 scope.go:117] "RemoveContainer" containerID="4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b"
Sep 18 23:06:20.664577 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:06:20.664560    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b\": container with ID starting with 4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b not found: ID does not exist" containerID="4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b"
Sep 18 23:06:20.664659 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.664588    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b"} err="failed to get container status \"4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b\": rpc error: code = NotFound desc = could not find container \"4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b\": container with ID starting with 4c222e3220c339f467f2f7ae02f6f7fddd9cf5fb46ba1c913900db0ef7a6bd2b not found: ID does not exist"
Sep 18 23:06:20.664659 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.664610    2582 scope.go:117] "RemoveContainer" containerID="3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d"
Sep 18 23:06:20.664869 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:06:20.664851    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d\": container with ID starting with 3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d not found: ID does not exist" containerID="3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d"
Sep 18 23:06:20.664974 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.664875    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d"} err="failed to get container status \"3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d\": rpc error: code = NotFound desc = could not find container \"3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d\": container with ID starting with 3226c0b6202a14a64c7bd0bf5809d4ebc63b358723e14cdacb7211dda6e0fc1d not found: ID does not exist"
Sep 18 23:06:20.669833 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.669810    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"]
Sep 18 23:06:20.673376 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.673354    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7bbf7dc599-pkcc7"]
Sep 18 23:06:20.974033 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:20.974002    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" path="/var/lib/kubelet/pods/0b81a169-f35b-4ff0-b79f-bf54cd607f92/volumes"
Sep 18 23:06:36.603314 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.603255    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"]
Sep 18 23:06:36.603793 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.603698    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main"
Sep 18 23:06:36.603793 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.603714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main"
Sep 18 23:06:36.603793 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.603739    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="storage-initializer"
Sep 18 23:06:36.603793 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.603745    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="storage-initializer"
Sep 18 23:06:36.603965 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.603807    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b81a169-f35b-4ff0-b79f-bf54cd607f92" containerName="main"
Sep 18 23:06:36.608418 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.608377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.612053 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.612015    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 18 23:06:36.613146 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.613120    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 18 23:06:36.613293 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.612020    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:06:36.613390 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.613356    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 18 23:06:36.613450 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.612065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-nltkt\""
Sep 18 23:06:36.613529 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.612041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 18 23:06:36.621718 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.621695    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"]
Sep 18 23:06:36.628177 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.628298 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.628298 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-model-cache\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.628298 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.628435 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-dshm\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.628435 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-home\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.628435 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.628579 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.628500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdc6s\" (UniqueName: \"kubernetes.io/projected/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kube-api-access-vdc6s\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729185 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-home\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729340 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729340 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdc6s\" (UniqueName: \"kubernetes.io/projected/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kube-api-access-vdc6s\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729428 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729428 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729532 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-model-cache\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729532 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729640 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729544    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-dshm\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729640 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-home\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729812 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729791    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729883 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-model-cache\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.729883 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.729848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.730211 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.730189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.731683 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.731659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-dshm\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.731970 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.731940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.736922 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.736904    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdc6s\" (UniqueName: \"kubernetes.io/projected/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kube-api-access-vdc6s\") pod \"lora-multiple-adapters-test-kserve-df694c4d4-ldl9m\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:36.923851 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:36.923814    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:37.204684 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:37.204596    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"]
Sep 18 23:06:37.207613 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:06:37.207584    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67186dd1_bfc5_4aab_a3e7_64cea7ed7fd6.slice/crio-a5e7a76f077d19912bff09f5bd4ac1ded9f05745149aff99dcc74dc7310adcb3 WatchSource:0}: Error finding container a5e7a76f077d19912bff09f5bd4ac1ded9f05745149aff99dcc74dc7310adcb3: Status 404 returned error can't find the container with id a5e7a76f077d19912bff09f5bd4ac1ded9f05745149aff99dcc74dc7310adcb3
Sep 18 23:06:37.737920 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:37.737872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" event={"ID":"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6","Type":"ContainerStarted","Data":"77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9"}
Sep 18 23:06:37.738386 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:37.737939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" event={"ID":"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6","Type":"ContainerStarted","Data":"a5e7a76f077d19912bff09f5bd4ac1ded9f05745149aff99dcc74dc7310adcb3"}
Sep 18 23:06:40.755402 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:40.755366    2582 generic.go:356] "Generic (PLEG): container finished" podID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerID="77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9" exitCode=0
Sep 18 23:06:40.755775 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:40.755410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" event={"ID":"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6","Type":"ContainerDied","Data":"77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9"}
Sep 18 23:06:41.763171 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:41.763118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" event={"ID":"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6","Type":"ContainerStarted","Data":"72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d"}
Sep 18 23:06:41.787180 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:41.787117    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podStartSLOduration=2.89760613 podStartE2EDuration="5.787095054s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.889488924s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:06:41.784158599 +0000 UTC m=+4558.817337492" watchObservedRunningTime="2026-09-18 23:06:41.787095054 +0000 UTC m=+4558.820273952"
Sep 18 23:06:41.924382 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:41.924342    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:46.924488 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:46.924449    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:06:46.926236 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:46.926205    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:06:56.925249 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:06:56.925201    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:07:06.925053 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:07:06.925002    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:07:16.924737 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:07:16.924693    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:07:26.924459 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:07:26.924406    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:07:36.925110 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:07:36.925064    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:07:46.924610 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:07:46.924558    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:07:56.935061 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:07:56.935029    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:07:56.942750 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:07:56.942725    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:08:37.465191 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:37.465122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-df694c4d4-ldl9m_67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6/storage-initializer/0.log"
Sep 18 23:08:37.474042 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:37.474016    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-df694c4d4-ldl9m_67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6/main/0.log"
Sep 18 23:08:38.506810 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:38.506256    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"]
Sep 18 23:08:53.531164 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:53.531121    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" containerID="cri-o://72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d" gracePeriod=45
Sep 18 23:08:53.935828 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:53.935785    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 18 23:08:54.187687 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.187630    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:08:54.322646 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-home\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.322646 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tmp-dir\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.322885 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-model-cache\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.322885 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322697    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tls-certs\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.322885 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdc6s\" (UniqueName: \"kubernetes.io/projected/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kube-api-access-vdc6s\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.322885 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322755    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-cabundle-cert\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.322885 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kserve-provision-location\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.323190 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.322896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-dshm\") pod \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\" (UID: \"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6\") "
Sep 18 23:08:54.323190 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.323000    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:54.323190 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.323138    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-home" (OuterVolumeSpecName: "home") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:54.323323 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.323208    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.323431 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.323388    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:08:54.325280 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.325252    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-dshm" (OuterVolumeSpecName: "dshm") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:54.325403 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.325384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kube-api-access-vdc6s" (OuterVolumeSpecName: "kube-api-access-vdc6s") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "kube-api-access-vdc6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:08:54.325447 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.325385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:08:54.341290 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.341255    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:54.351000 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.350976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" (UID: "67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:08:54.423895 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.423858    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.423895 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.423885    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.423895 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.423901    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.424183 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.423912    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.424183 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.423920    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdc6s\" (UniqueName: \"kubernetes.io/projected/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kube-api-access-vdc6s\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.424183 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.423929    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.424183 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.423937    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:08:54.531412 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.531317    2582 generic.go:356] "Generic (PLEG): container finished" podID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerID="72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d" exitCode=0
Sep 18 23:08:54.531412 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.531382    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"
Sep 18 23:08:54.531412 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.531400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" event={"ID":"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6","Type":"ContainerDied","Data":"72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d"}
Sep 18 23:08:54.532024 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.531456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m" event={"ID":"67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6","Type":"ContainerDied","Data":"a5e7a76f077d19912bff09f5bd4ac1ded9f05745149aff99dcc74dc7310adcb3"}
Sep 18 23:08:54.532024 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.531478    2582 scope.go:117] "RemoveContainer" containerID="72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d"
Sep 18 23:08:54.541418 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.541255    2582 scope.go:117] "RemoveContainer" containerID="77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9"
Sep 18 23:08:54.550174 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.550151    2582 scope.go:117] "RemoveContainer" containerID="72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d"
Sep 18 23:08:54.550436 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:08:54.550414    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d\": container with ID starting with 72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d not found: ID does not exist" containerID="72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d"
Sep 18 23:08:54.550519 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.550446    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d"} err="failed to get container status \"72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d\": rpc error: code = NotFound desc = could not find container \"72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d\": container with ID starting with 72e51e5e585f53a82a325f6e6f472b5847d0fabc83e7995a3c827cc9c47dd16d not found: ID does not exist"
Sep 18 23:08:54.550519 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.550470    2582 scope.go:117] "RemoveContainer" containerID="77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9"
Sep 18 23:08:54.550732 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:08:54.550708    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9\": container with ID starting with 77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9 not found: ID does not exist" containerID="77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9"
Sep 18 23:08:54.550795 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.550738    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9"} err="failed to get container status \"77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9\": rpc error: code = NotFound desc = could not find container \"77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9\": container with ID starting with 77001eb196c5ac9b1a5b3ea3555d6472364501e8d0e338aaeb611b898fa7b7b9 not found: ID does not exist"
Sep 18 23:08:54.554892 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.554869    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"]
Sep 18 23:08:54.560065 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.560042    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-df694c4d4-ldl9m"]
Sep 18 23:08:54.974166 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:08:54.974134    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" path="/var/lib/kubelet/pods/67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6/volumes"
Sep 18 23:09:01.920231 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.920197    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"]
Sep 18 23:09:01.920625 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.920561    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="storage-initializer"
Sep 18 23:09:01.920625 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.920573    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="storage-initializer"
Sep 18 23:09:01.920625 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.920594    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main"
Sep 18 23:09:01.920625 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.920600    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main"
Sep 18 23:09:01.920758 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.920654    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67186dd1-bfc5-4aab-a3e7-64cea7ed7fd6" containerName="main"
Sep 18 23:09:01.923762 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.923746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:01.926098 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.926077    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-ws7d4\""
Sep 18 23:09:01.926221 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.926081    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 23:09:01.926808 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.926787    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 23:09:01.926808 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.926799    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 23:09:01.930777 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.930756    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"]
Sep 18 23:09:01.990223 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:01.990190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46h8q\" (UniqueName: \"kubernetes.io/projected/038ba606-b613-4b72-8d58-5c328e12046f-kube-api-access-46h8q\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:02.091192 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:02.091159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46h8q\" (UniqueName: \"kubernetes.io/projected/038ba606-b613-4b72-8d58-5c328e12046f-kube-api-access-46h8q\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:02.100010 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:02.099984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46h8q\" (UniqueName: \"kubernetes.io/projected/038ba606-b613-4b72-8d58-5c328e12046f-kube-api-access-46h8q\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:09.861177 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:09.861067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:09.866738 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:09.866715    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:09.962184 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:09.962150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:09.962366 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:09.962328    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:09.965261 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:09.965239    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") DevicePath \"csi-2f2dae96cc6e898858c2369e2b1825f15f66d31cfd84235ad13e88e60c604206\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:10.292505 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:10.292468    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d0f3c13a67210d12dbee5e7e2d01b303f2b0d8b33caa5fe184c612fbf6abd2e/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:10.307357 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:10.307330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"e2e-pvc-lora-download-fgnkr\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:10.334098 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:10.334068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:10.607221 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:10.607195    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"]
Sep 18 23:09:10.609444 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:09:10.609407    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod038ba606_b613_4b72_8d58_5c328e12046f.slice/crio-ae3ccc5061cdc71433155d0ce97e11323be9c45843a5aec5f806e8430a2a6550 WatchSource:0}: Error finding container ae3ccc5061cdc71433155d0ce97e11323be9c45843a5aec5f806e8430a2a6550: Status 404 returned error can't find the container with id ae3ccc5061cdc71433155d0ce97e11323be9c45843a5aec5f806e8430a2a6550
Sep 18 23:09:10.618026 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:10.617992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr" event={"ID":"038ba606-b613-4b72-8d58-5c328e12046f","Type":"ContainerStarted","Data":"ae3ccc5061cdc71433155d0ce97e11323be9c45843a5aec5f806e8430a2a6550"}
Sep 18 23:09:11.628606 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:11.628560    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr" event={"ID":"038ba606-b613-4b72-8d58-5c328e12046f","Type":"ContainerStarted","Data":"4b874dc3deb72231b31cb1182a66a5f74c9cf897163dc47eba5c9a270796c612"}
Sep 18 23:09:11.645734 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:11.645677    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr" podStartSLOduration=13.645661076 podStartE2EDuration="13.645661076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 23:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:09:11.644182137 +0000 UTC m=+4708.677361044" watchObservedRunningTime="2026-09-18 23:09:11.645661076 +0000 UTC m=+4708.678839996"
Sep 18 23:09:13.640607 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:13.640573    2582 generic.go:356] "Generic (PLEG): container finished" podID="038ba606-b613-4b72-8d58-5c328e12046f" containerID="4b874dc3deb72231b31cb1182a66a5f74c9cf897163dc47eba5c9a270796c612" exitCode=0
Sep 18 23:09:13.640991 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:13.640649    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr" event={"ID":"038ba606-b613-4b72-8d58-5c328e12046f","Type":"ContainerDied","Data":"4b874dc3deb72231b31cb1182a66a5f74c9cf897163dc47eba5c9a270796c612"}
Sep 18 23:09:14.777022 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:14.776993    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:14.905845 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:14.905750    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46h8q\" (UniqueName: \"kubernetes.io/projected/038ba606-b613-4b72-8d58-5c328e12046f-kube-api-access-46h8q\") pod \"038ba606-b613-4b72-8d58-5c328e12046f\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") "
Sep 18 23:09:14.906044 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:14.905886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"038ba606-b613-4b72-8d58-5c328e12046f\" (UID: \"038ba606-b613-4b72-8d58-5c328e12046f\") "
Sep 18 23:09:14.908212 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:14.908185    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038ba606-b613-4b72-8d58-5c328e12046f-kube-api-access-46h8q" (OuterVolumeSpecName: "kube-api-access-46h8q") pod "038ba606-b613-4b72-8d58-5c328e12046f" (UID: "038ba606-b613-4b72-8d58-5c328e12046f"). InnerVolumeSpecName "kube-api-access-46h8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:09:14.909927 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:14.909904    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b" (OuterVolumeSpecName: "model-storage") pod "038ba606-b613-4b72-8d58-5c328e12046f" (UID: "038ba606-b613-4b72-8d58-5c328e12046f"). InnerVolumeSpecName "pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 23:09:15.007323 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:15.007299    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46h8q\" (UniqueName: \"kubernetes.io/projected/038ba606-b613-4b72-8d58-5c328e12046f-kube-api-access-46h8q\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:09:15.007430 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:15.007340    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") on node \"ip-10-0-142-209.ec2.internal\" "
Sep 18 23:09:15.651716 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:15.651680    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"
Sep 18 23:09:15.651716 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:15.651712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr" event={"ID":"038ba606-b613-4b72-8d58-5c328e12046f","Type":"ContainerDied","Data":"ae3ccc5061cdc71433155d0ce97e11323be9c45843a5aec5f806e8430a2a6550"}
Sep 18 23:09:15.651936 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:15.651747    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3ccc5061cdc71433155d0ce97e11323be9c45843a5aec5f806e8430a2a6550"
Sep 18 23:09:15.982427 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:15.982397    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b") on node "ip-10-0-142-209.ec2.internal"
Sep 18 23:09:16.015224 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:16.015197    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"csi-2f2dae96cc6e898858c2369e2b1825f15f66d31cfd84235ad13e88e60c604206\""
Sep 18 23:09:45.744310 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.744275    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"]
Sep 18 23:09:45.745143 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.744820    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="038ba606-b613-4b72-8d58-5c328e12046f" containerName="storage-initializer"
Sep 18 23:09:45.745143 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.744842    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="038ba606-b613-4b72-8d58-5c328e12046f" containerName="storage-initializer"
Sep 18 23:09:45.745143 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.744972    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="038ba606-b613-4b72-8d58-5c328e12046f" containerName="storage-initializer"
Sep 18 23:09:45.748434 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.748410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.751618 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.751591    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Sep 18 23:09:45.751746 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.751722    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Sep 18 23:09:45.751822 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.751781    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:09:45.751822 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.751785    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-pdcfl\""
Sep 18 23:09:45.752069 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.752051    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 23:09:45.752297 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.752282    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Sep 18 23:09:45.752382 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.752284    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-dockercfg-jfflg\""
Sep 18 23:09:45.753771 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.753745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-kserve-provision-location\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.753870 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.753798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-tmp-dir\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.753870 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.753823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5318e729-a528-4255-87ad-f1080b6ffe7d-tls-certs\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.753992 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.753874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-model-cache\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.753992 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.753902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnfx\" (UniqueName: \"kubernetes.io/projected/5318e729-a528-4255-87ad-f1080b6ffe7d-kube-api-access-mjnfx\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.753992 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.753962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-home\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.754127 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.754005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-dshm\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.754127 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.754053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5318e729-a528-4255-87ad-f1080b6ffe7d-cabundle-cert\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.768656 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.768630    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"]
Sep 18 23:09:45.855004 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.854940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5318e729-a528-4255-87ad-f1080b6ffe7d-cabundle-cert\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855202 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-kserve-provision-location\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855202 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-tmp-dir\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855202 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5318e729-a528-4255-87ad-f1080b6ffe7d-tls-certs\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855386 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-model-cache\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855386 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnfx\" (UniqueName: \"kubernetes.io/projected/5318e729-a528-4255-87ad-f1080b6ffe7d-kube-api-access-mjnfx\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855386 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-home\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855386 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-dshm\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855607 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-kserve-provision-location\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855607 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-tmp-dir\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855607 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-model-cache\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855760 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-home\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.855760 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.855726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5318e729-a528-4255-87ad-f1080b6ffe7d-cabundle-cert\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.857535 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.857505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-dshm\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.857786 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.857766    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5318e729-a528-4255-87ad-f1080b6ffe7d-tls-certs\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:45.863412 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:45.863393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnfx\" (UniqueName: \"kubernetes.io/projected/5318e729-a528-4255-87ad-f1080b6ffe7d-kube-api-access-mjnfx\") pod \"tls-verification-test-kserve-56b598698f-4h4fv\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:46.059356 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:46.059267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:46.355862 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:46.355829    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"]
Sep 18 23:09:46.357799 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:09:46.357771    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5318e729_a528_4255_87ad_f1080b6ffe7d.slice/crio-699aff9a6d32890999b550fc93b5a8a669946ff4ede372b593088403e340425e WatchSource:0}: Error finding container 699aff9a6d32890999b550fc93b5a8a669946ff4ede372b593088403e340425e: Status 404 returned error can't find the container with id 699aff9a6d32890999b550fc93b5a8a669946ff4ede372b593088403e340425e
Sep 18 23:09:46.810771 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:46.810732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerStarted","Data":"915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80"}
Sep 18 23:09:46.811196 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:46.810791    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerStarted","Data":"699aff9a6d32890999b550fc93b5a8a669946ff4ede372b593088403e340425e"}
Sep 18 23:09:46.811196 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:46.810958    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:47.821825 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:47.821782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerStarted","Data":"edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5"}
Sep 18 23:09:49.834626 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:49.834590    2582 generic.go:356] "Generic (PLEG): container finished" podID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerID="edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5" exitCode=0
Sep 18 23:09:49.835051 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:49.834664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerDied","Data":"edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5"}
Sep 18 23:09:50.842338 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:50.842297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerStarted","Data":"3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e"}
Sep 18 23:09:50.873027 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:50.872970    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podStartSLOduration=3.669436375 podStartE2EDuration="5.872938901s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.203502526s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:09:50.87209687 +0000 UTC m=+4747.905275765" watchObservedRunningTime="2026-09-18 23:09:50.872938901 +0000 UTC m=+4747.906117796"
Sep 18 23:09:56.059829 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:56.059789    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:56.060405 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:56.059859    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:09:56.061244 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:56.061210    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:09:56.072721 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:09:56.072697    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:10:06.060476 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:06.060428    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:10:08.721194 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.721152    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"]
Sep 18 23:10:08.728713 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.728678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.733537 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.732746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 23:10:08.733537 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.732992    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-ws7d4\""
Sep 18 23:10:08.733975 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.733887    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 18 23:10:08.734734 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.734570    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 23:10:08.734850 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.734814    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 23:10:08.735053 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.735026    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:10:08.739192 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.738550    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"]
Sep 18 23:10:08.864791 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.864750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-dshm\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.865021 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.864805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4cm9\" (UniqueName: \"kubernetes.io/projected/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kube-api-access-f4cm9\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.865021 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.864832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.865021 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.864862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.865021 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.864892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-home\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.865021 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.864996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.865239 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.865045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.865239 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.865111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.965707 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.965674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4cm9\" (UniqueName: \"kubernetes.io/projected/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kube-api-access-f4cm9\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.965882 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.965717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.965882 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.965742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.965882 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.965783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-home\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.965882 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.965817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.965882 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.965847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.966203 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.965904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.966203 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.966049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-dshm\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.966203 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.966179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.966387 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.966358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.966518 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.966420    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-home\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.966518 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.966486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.966739 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.966718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.968650 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.968629    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-dshm\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.968977 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.968938    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:08.974066 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:08.974006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4cm9\" (UniqueName: \"kubernetes.io/projected/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kube-api-access-f4cm9\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.084676 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.084626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.090176 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.090149    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.186202 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.186161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.186365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.186281    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.193062 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.193041    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") DevicePath \"csi-2f2dae96cc6e898858c2369e2b1825f15f66d31cfd84235ad13e88e60c604206\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.281229 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.281188    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d0f3c13a67210d12dbee5e7e2d01b303f2b0d8b33caa5fe184c612fbf6abd2e/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.301443 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.301410    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.444571 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.444515    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:11.729929 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.729900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"]
Sep 18 23:10:11.732052 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:10:11.732018    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f38b20d_74a2_4df6_a878_a1b6fba8ee5f.slice/crio-bfae9c0b5539998272474553d33d1fd1d0c251b201a9fe7454cd7198ff9ede34 WatchSource:0}: Error finding container bfae9c0b5539998272474553d33d1fd1d0c251b201a9fe7454cd7198ff9ede34: Status 404 returned error can't find the container with id bfae9c0b5539998272474553d33d1fd1d0c251b201a9fe7454cd7198ff9ede34
Sep 18 23:10:11.963039 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.963001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" event={"ID":"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f","Type":"ContainerStarted","Data":"ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d"}
Sep 18 23:10:11.963228 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:11.963056    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" event={"ID":"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f","Type":"ContainerStarted","Data":"bfae9c0b5539998272474553d33d1fd1d0c251b201a9fe7454cd7198ff9ede34"}
Sep 18 23:10:14.981475 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:14.981437    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerID="ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d" exitCode=0
Sep 18 23:10:14.981854 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:14.981508    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" event={"ID":"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f","Type":"ContainerDied","Data":"ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d"}
Sep 18 23:10:15.991102 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:15.991065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" event={"ID":"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f","Type":"ContainerStarted","Data":"5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a"}
Sep 18 23:10:16.024167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:16.024107    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podStartSLOduration=5.659552552 podStartE2EDuration="8.024089975s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.364537423s" isStatefulPod=true podCreationTimestamp="2026-09-18 23:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:10:16.022123304 +0000 UTC m=+4773.055302199" watchObservedRunningTime="2026-09-18 23:10:16.024089975 +0000 UTC m=+4773.057268870"
Sep 18 23:10:16.060395 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:16.060351    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:10:16.445595 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:16.445557    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:21.445612 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:21.445564    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:10:21.447117 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:21.447083    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:10:26.060364 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:26.060306    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:10:31.445033 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:31.444982    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:10:36.060787 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:36.060678    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:10:41.445316 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:41.445257    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:10:45.474608 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:45.474579    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:10:45.480933 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:45.480907    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:10:45.491615 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:45.491595    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:10:45.497742 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:45.497720    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:10:46.059890 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:46.059837    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:10:51.445846 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:51.445789    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:10:56.059923 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:10:56.059841    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:11:01.446131 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:01.446074    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:11:06.060009 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:06.059933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:11:11.445231 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:11.445170    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:11:16.060047 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:16.059990    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:11:21.445359 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:21.445306    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:11:26.060408 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:26.060357    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:11:31.454884 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:31.454788    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:11:31.462427 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:31.462398    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:11:36.059795 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:36.059750    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:11:46.060462 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:46.060402    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:11:56.059830 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:11:56.059787    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 18 23:12:06.069607 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:06.069574    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:12:06.085872 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:06.085845    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:12:09.875052 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:09.875022    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh_3f38b20d-74a2-4df6-a878-a1b6fba8ee5f/storage-initializer/0.log"
Sep 18 23:12:09.884016 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:09.883988    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh_3f38b20d-74a2-4df6-a878-a1b6fba8ee5f/main/0.log"
Sep 18 23:12:10.909807 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:10.909771    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"]
Sep 18 23:12:25.933924 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:25.933861    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" containerID="cri-o://5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a" gracePeriod=45
Sep 18 23:12:26.455962 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.455906    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 18 23:12:26.593221 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.593196    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:12:26.629290 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629221    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4cm9\" (UniqueName: \"kubernetes.io/projected/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kube-api-access-f4cm9\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629290 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629273    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-dshm\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629475 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629355    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629520 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629487    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kserve-provision-location\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629569 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tmp-dir\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629627 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629581    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-model-cache\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629627 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629608    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-cabundle-cert\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629743 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tls-certs\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629743 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629680    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-home\") pod \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\" (UID: \"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f\") "
Sep 18 23:12:26.629909 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.629872    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:12:26.630269 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.630244    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:12:26.630580 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.630542    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-home" (OuterVolumeSpecName: "home") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:12:26.632468 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.632441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kube-api-access-f4cm9" (OuterVolumeSpecName: "kube-api-access-f4cm9") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "kube-api-access-f4cm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:12:26.632802 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.632769    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-dshm" (OuterVolumeSpecName: "dshm") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:12:26.632882 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.632831    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:12:26.635960 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.635921    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 23:12:26.638408 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.638383    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:12:26.644921 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.644893    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" (UID: "3f38b20d-74a2-4df6-a878-a1b6fba8ee5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:12:26.731188 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731157    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.731188 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731184    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4cm9\" (UniqueName: \"kubernetes.io/projected/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kube-api-access-f4cm9\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.731365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731197    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.731365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731225    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") on node \"ip-10-0-142-209.ec2.internal\" "
Sep 18 23:12:26.731365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731235    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.731365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731244    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.731365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731252    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.731365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731260    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.731365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.731267    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:26.741550 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.741532    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b") on node "ip-10-0-142-209.ec2.internal"
Sep 18 23:12:26.788333 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.788302    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerID="5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a" exitCode=0
Sep 18 23:12:26.788462 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.788355    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"
Sep 18 23:12:26.788462 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.788387    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" event={"ID":"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f","Type":"ContainerDied","Data":"5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a"}
Sep 18 23:12:26.788462 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.788438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh" event={"ID":"3f38b20d-74a2-4df6-a878-a1b6fba8ee5f","Type":"ContainerDied","Data":"bfae9c0b5539998272474553d33d1fd1d0c251b201a9fe7454cd7198ff9ede34"}
Sep 18 23:12:26.788462 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.788459    2582 scope.go:117] "RemoveContainer" containerID="5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a"
Sep 18 23:12:26.798936 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.798920    2582 scope.go:117] "RemoveContainer" containerID="ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d"
Sep 18 23:12:26.807313 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.807293    2582 scope.go:117] "RemoveContainer" containerID="5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a"
Sep 18 23:12:26.807570 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:12:26.807551    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a\": container with ID starting with 5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a not found: ID does not exist" containerID="5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a"
Sep 18 23:12:26.807642 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.807577    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a"} err="failed to get container status \"5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a\": rpc error: code = NotFound desc = could not find container \"5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a\": container with ID starting with 5b8a2ef60552edcc0926654b73c49d931b430119893081f5012a967e6810f79a not found: ID does not exist"
Sep 18 23:12:26.807642 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.807600    2582 scope.go:117] "RemoveContainer" containerID="ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d"
Sep 18 23:12:26.807829 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:12:26.807814    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d\": container with ID starting with ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d not found: ID does not exist" containerID="ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d"
Sep 18 23:12:26.808025 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.807832    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d"} err="failed to get container status \"ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d\": rpc error: code = NotFound desc = could not find container \"ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d\": container with ID starting with ca416379689af561ae8a028ce87a41e2fe454e14168e0657b102cb1483598d4d not found: ID does not exist"
Sep 18 23:12:26.812930 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.812907    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"]
Sep 18 23:12:26.816476 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.816449    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-666f77f7c9-g87kh"]
Sep 18 23:12:26.831883 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.831857    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-235945ae-6a25-477f-91bc-5eb2e19d50f3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0764959610717294b\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"csi-2f2dae96cc6e898858c2369e2b1825f15f66d31cfd84235ad13e88e60c604206\""
Sep 18 23:12:26.974518 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:26.974438    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" path="/var/lib/kubelet/pods/3f38b20d-74a2-4df6-a878-a1b6fba8ee5f/volumes"
Sep 18 23:12:34.526243 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.526202    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"]
Sep 18 23:12:34.526631 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.526609    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main"
Sep 18 23:12:34.526631 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.526620    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main"
Sep 18 23:12:34.526704 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.526637    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="storage-initializer"
Sep 18 23:12:34.526704 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.526645    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="storage-initializer"
Sep 18 23:12:34.526769 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.526706    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f38b20d-74a2-4df6-a878-a1b6fba8ee5f" containerName="main"
Sep 18 23:12:34.529774 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.529753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:34.532134 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.532109    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 18 23:12:34.532265 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.532143    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 23:12:34.532265 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.532176    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 23:12:34.532265 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.532188    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-9nwvf\""
Sep 18 23:12:34.537964 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.537924    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"]
Sep 18 23:12:34.601064 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.601029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8vw\" (UniqueName: \"kubernetes.io/projected/8ba00827-42d7-445b-a217-9b3f6a64ff7d-kube-api-access-sw8vw\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:34.702453 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.702406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8vw\" (UniqueName: \"kubernetes.io/projected/8ba00827-42d7-445b-a217-9b3f6a64ff7d-kube-api-access-sw8vw\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:34.711097 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:34.711070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8vw\" (UniqueName: \"kubernetes.io/projected/8ba00827-42d7-445b-a217-9b3f6a64ff7d-kube-api-access-sw8vw\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:36.207839 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:36.207794    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"]
Sep 18 23:12:36.214393 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:36.214365    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-fgnkr"]
Sep 18 23:12:36.974712 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:36.974676    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="038ba606-b613-4b72-8d58-5c328e12046f" path="/var/lib/kubelet/pods/038ba606-b613-4b72-8d58-5c328e12046f/volumes"
Sep 18 23:12:44.385235 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:44.385201    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:44.390261 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:44.390233    2582 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:44.486101 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:44.486057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:44.486271 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:44.486196    2582 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:44.488927 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:44.488907    2582 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") DevicePath \"csi-522d1401dfb152ed187b99e35e0e8d56a7f5f3d62a660e5ab3f7cf9976863a8c\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:44.768002 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:44.767965    2582 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9f88c7dcb205a4ab1e755ef598d6ac0d3322c3e055fcdfe662aa08fc65c6ffbd/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:44.781617 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:44.781591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"e2e-pvc-lora-download-ltfzq\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:45.041976 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:45.041879    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-9nwvf\""
Sep 18 23:12:45.049874 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:45.049850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:45.323675 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:45.323649    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"]
Sep 18 23:12:45.325734 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:12:45.325705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ba00827_42d7_445b_a217_9b3f6a64ff7d.slice/crio-fa88dce714d48ec7eba7d81d17755b72910b2f1c77db96a613ae6478d8432614 WatchSource:0}: Error finding container fa88dce714d48ec7eba7d81d17755b72910b2f1c77db96a613ae6478d8432614: Status 404 returned error can't find the container with id fa88dce714d48ec7eba7d81d17755b72910b2f1c77db96a613ae6478d8432614
Sep 18 23:12:45.329760 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:45.329734    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 18 23:12:45.896051 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:45.896006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq" event={"ID":"8ba00827-42d7-445b-a217-9b3f6a64ff7d","Type":"ContainerStarted","Data":"fb9338dc3536f67eb243543846c7bfc6c1c7a75e9374405fe0688601aed5f1f4"}
Sep 18 23:12:45.896471 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:45.896059    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq" event={"ID":"8ba00827-42d7-445b-a217-9b3f6a64ff7d","Type":"ContainerStarted","Data":"fa88dce714d48ec7eba7d81d17755b72910b2f1c77db96a613ae6478d8432614"}
Sep 18 23:12:45.915761 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:45.915699    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq" podStartSLOduration=14.915679349 podStartE2EDuration="14.915679349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 23:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:12:45.91172285 +0000 UTC m=+4922.944901758" watchObservedRunningTime="2026-09-18 23:12:45.915679349 +0000 UTC m=+4922.948858243"
Sep 18 23:12:47.907292 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:47.907260    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ba00827-42d7-445b-a217-9b3f6a64ff7d" containerID="fb9338dc3536f67eb243543846c7bfc6c1c7a75e9374405fe0688601aed5f1f4" exitCode=0
Sep 18 23:12:47.907658 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:47.907332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq" event={"ID":"8ba00827-42d7-445b-a217-9b3f6a64ff7d","Type":"ContainerDied","Data":"fb9338dc3536f67eb243543846c7bfc6c1c7a75e9374405fe0688601aed5f1f4"}
Sep 18 23:12:49.063493 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.063462    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:49.123170 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.123136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") pod \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") "
Sep 18 23:12:49.123349 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.123239    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw8vw\" (UniqueName: \"kubernetes.io/projected/8ba00827-42d7-445b-a217-9b3f6a64ff7d-kube-api-access-sw8vw\") pod \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\" (UID: \"8ba00827-42d7-445b-a217-9b3f6a64ff7d\") "
Sep 18 23:12:49.125534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.125498    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba00827-42d7-445b-a217-9b3f6a64ff7d-kube-api-access-sw8vw" (OuterVolumeSpecName: "kube-api-access-sw8vw") pod "8ba00827-42d7-445b-a217-9b3f6a64ff7d" (UID: "8ba00827-42d7-445b-a217-9b3f6a64ff7d"). InnerVolumeSpecName "kube-api-access-sw8vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:12:49.125840 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.125821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434" (OuterVolumeSpecName: "model-storage") pod "8ba00827-42d7-445b-a217-9b3f6a64ff7d" (UID: "8ba00827-42d7-445b-a217-9b3f6a64ff7d"). InnerVolumeSpecName "pvc-56b90f26-d636-4f5c-84b8-a352543a0b05". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 23:12:49.224578 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.224484    2582 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") on node \"ip-10-0-142-209.ec2.internal\" "
Sep 18 23:12:49.224578 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.224525    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw8vw\" (UniqueName: \"kubernetes.io/projected/8ba00827-42d7-445b-a217-9b3f6a64ff7d-kube-api-access-sw8vw\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:12:49.919617 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.919590    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"
Sep 18 23:12:49.919793 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.919626    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq" event={"ID":"8ba00827-42d7-445b-a217-9b3f6a64ff7d","Type":"ContainerDied","Data":"fa88dce714d48ec7eba7d81d17755b72910b2f1c77db96a613ae6478d8432614"}
Sep 18 23:12:49.919793 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:49.919666    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa88dce714d48ec7eba7d81d17755b72910b2f1c77db96a613ae6478d8432614"
Sep 18 23:12:50.195717 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:50.195697    2582 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-56b90f26-d636-4f5c-84b8-a352543a0b05" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434") on node "ip-10-0-142-209.ec2.internal"
Sep 18 23:12:50.232103 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:12:50.232069    2582 reconciler_common.go:299] "Volume detached for volume \"pvc-56b90f26-d636-4f5c-84b8-a352543a0b05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e1556e42d3788434\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"csi-522d1401dfb152ed187b99e35e0e8d56a7f5f3d62a660e5ab3f7cf9976863a8c\""
Sep 18 23:15:23.321118 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:23.321072    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"]
Sep 18 23:15:23.326739 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:23.326709    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-ltfzq"]
Sep 18 23:15:24.974405 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:24.974369    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba00827-42d7-445b-a217-9b3f6a64ff7d" path="/var/lib/kubelet/pods/8ba00827-42d7-445b-a217-9b3f6a64ff7d/volumes"
Sep 18 23:15:45.516258 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:45.516227    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:15:45.521633 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:45.521612    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:15:45.534742 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:45.534722    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:15:45.540101 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:45.540084    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:15:53.240345 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:15:53.240309    2582 scope.go:117] "RemoveContainer" containerID="4b874dc3deb72231b31cb1182a66a5f74c9cf897163dc47eba5c9a270796c612"
Sep 18 23:16:18.041172 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.041086    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"]
Sep 18 23:16:18.041636 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.041479    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba00827-42d7-445b-a217-9b3f6a64ff7d" containerName="storage-initializer"
Sep 18 23:16:18.041636 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.041495    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba00827-42d7-445b-a217-9b3f6a64ff7d" containerName="storage-initializer"
Sep 18 23:16:18.041636 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.041565    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba00827-42d7-445b-a217-9b3f6a64ff7d" containerName="storage-initializer"
Sep 18 23:16:18.044526 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.044508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.047665 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.047644    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 23:16:18.047813 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.047645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-8hsv8\""
Sep 18 23:16:18.047813 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.047645    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 23:16:18.047813 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.047675    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 23:16:18.053332 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.053308    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"]
Sep 18 23:16:18.136753 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.136717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.136930 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.136756    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhlx\" (UniqueName: \"kubernetes.io/projected/0da600b2-4065-42ed-a4e4-d388f49384f8-kube-api-access-8dhlx\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.136930 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.136792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.136930 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.136862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.136930 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.136904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da600b2-4065-42ed-a4e4-d388f49384f8-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.136930 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.136926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238321 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238321 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhlx\" (UniqueName: \"kubernetes.io/projected/0da600b2-4065-42ed-a4e4-d388f49384f8-kube-api-access-8dhlx\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238576 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238576 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238576 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da600b2-4065-42ed-a4e4-d388f49384f8-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238576 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238779 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238838 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.238838 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.238825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.240806 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.240773    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.241037 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.241020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da600b2-4065-42ed-a4e4-d388f49384f8-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.246245 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.246223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhlx\" (UniqueName: \"kubernetes.io/projected/0da600b2-4065-42ed-a4e4-d388f49384f8-kube-api-access-8dhlx\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4vt82\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.355781 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.355691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:18.630556 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:18.630523    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"]
Sep 18 23:16:18.632475 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:16:18.632450    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da600b2_4065_42ed_a4e4_d388f49384f8.slice/crio-1f8ed1be2f523f4de2d87e98fecef9cb37d8684e4154836897db3c2086089693 WatchSource:0}: Error finding container 1f8ed1be2f523f4de2d87e98fecef9cb37d8684e4154836897db3c2086089693: Status 404 returned error can't find the container with id 1f8ed1be2f523f4de2d87e98fecef9cb37d8684e4154836897db3c2086089693
Sep 18 23:16:19.066849 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:19.066755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" event={"ID":"0da600b2-4065-42ed-a4e4-d388f49384f8","Type":"ContainerStarted","Data":"488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757"}
Sep 18 23:16:19.066849 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:19.066818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" event={"ID":"0da600b2-4065-42ed-a4e4-d388f49384f8","Type":"ContainerStarted","Data":"1f8ed1be2f523f4de2d87e98fecef9cb37d8684e4154836897db3c2086089693"}
Sep 18 23:16:19.088287 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:19.088232    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" podStartSLOduration=1.088217155 podStartE2EDuration="1.088217155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:16:19.086673564 +0000 UTC m=+5136.119852461" watchObservedRunningTime="2026-09-18 23:16:19.088217155 +0000 UTC m=+5136.121396112"
Sep 18 23:16:19.356600 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:19.356504    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:21.585443 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:21.585406    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"]
Sep 18 23:16:22.091623 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:16:22.091585    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 23:16:22.091623 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:16:22.091623 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 23:16:22.091623 ip-10-0-142-209 kubenswrapper[2582]:  > containerID="488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757" cmd=["/bin/sleep","15"]
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.091634    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" message="" err=<
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]:  >
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:16:22.091647    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" podUID="0da600b2-4065-42ed-a4e4-d388f49384f8" containerName="main" containerID="cri-o://488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757"
Sep 18 23:16:22.091914 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.091689    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" podUID="0da600b2-4065-42ed-a4e4-d388f49384f8" containerName="main" containerID="cri-o://488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757" gracePeriod=60
Sep 18 23:16:22.344433 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.344370    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:22.475980 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.475928    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-dshm\") pod \"0da600b2-4065-42ed-a4e4-d388f49384f8\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") "
Sep 18 23:16:22.475980 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.475984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dhlx\" (UniqueName: \"kubernetes.io/projected/0da600b2-4065-42ed-a4e4-d388f49384f8-kube-api-access-8dhlx\") pod \"0da600b2-4065-42ed-a4e4-d388f49384f8\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") "
Sep 18 23:16:22.476251 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476009    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-tmp-dir\") pod \"0da600b2-4065-42ed-a4e4-d388f49384f8\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") "
Sep 18 23:16:22.476251 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476068    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-home\") pod \"0da600b2-4065-42ed-a4e4-d388f49384f8\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") "
Sep 18 23:16:22.476251 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476129    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da600b2-4065-42ed-a4e4-d388f49384f8-tls-certs\") pod \"0da600b2-4065-42ed-a4e4-d388f49384f8\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") "
Sep 18 23:16:22.476251 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-model-cache\") pod \"0da600b2-4065-42ed-a4e4-d388f49384f8\" (UID: \"0da600b2-4065-42ed-a4e4-d388f49384f8\") "
Sep 18 23:16:22.476475 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476291    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0da600b2-4065-42ed-a4e4-d388f49384f8" (UID: "0da600b2-4065-42ed-a4e4-d388f49384f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:22.476475 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476374    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-home" (OuterVolumeSpecName: "home") pod "0da600b2-4065-42ed-a4e4-d388f49384f8" (UID: "0da600b2-4065-42ed-a4e4-d388f49384f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:22.476596 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476484    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.476596 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476501    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.476596 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.476574    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "0da600b2-4065-42ed-a4e4-d388f49384f8" (UID: "0da600b2-4065-42ed-a4e4-d388f49384f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:22.478280 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.478260    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-dshm" (OuterVolumeSpecName: "dshm") pod "0da600b2-4065-42ed-a4e4-d388f49384f8" (UID: "0da600b2-4065-42ed-a4e4-d388f49384f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:16:22.478378 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.478306    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da600b2-4065-42ed-a4e4-d388f49384f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0da600b2-4065-42ed-a4e4-d388f49384f8" (UID: "0da600b2-4065-42ed-a4e4-d388f49384f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:16:22.478437 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.478406    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da600b2-4065-42ed-a4e4-d388f49384f8-kube-api-access-8dhlx" (OuterVolumeSpecName: "kube-api-access-8dhlx") pod "0da600b2-4065-42ed-a4e4-d388f49384f8" (UID: "0da600b2-4065-42ed-a4e4-d388f49384f8"). InnerVolumeSpecName "kube-api-access-8dhlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:16:22.577554 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.577521    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0da600b2-4065-42ed-a4e4-d388f49384f8-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.577554 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.577549    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.577554 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.577557    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0da600b2-4065-42ed-a4e4-d388f49384f8-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:16:22.577812 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:22.577565    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dhlx\" (UniqueName: \"kubernetes.io/projected/0da600b2-4065-42ed-a4e4-d388f49384f8-kube-api-access-8dhlx\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:16:23.088743 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.088704    2582 generic.go:356] "Generic (PLEG): container finished" podID="0da600b2-4065-42ed-a4e4-d388f49384f8" containerID="488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757" exitCode=0
Sep 18 23:16:23.089167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.088754    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"
Sep 18 23:16:23.089167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.088780    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" event={"ID":"0da600b2-4065-42ed-a4e4-d388f49384f8","Type":"ContainerDied","Data":"488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757"}
Sep 18 23:16:23.089167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.088831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82" event={"ID":"0da600b2-4065-42ed-a4e4-d388f49384f8","Type":"ContainerDied","Data":"1f8ed1be2f523f4de2d87e98fecef9cb37d8684e4154836897db3c2086089693"}
Sep 18 23:16:23.089167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.088855    2582 scope.go:117] "RemoveContainer" containerID="488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757"
Sep 18 23:16:23.098369 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.098088    2582 scope.go:117] "RemoveContainer" containerID="488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757"
Sep 18 23:16:23.098369 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:16:23.098347    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757\": container with ID starting with 488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757 not found: ID does not exist" containerID="488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757"
Sep 18 23:16:23.098464 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.098370    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757"} err="failed to get container status \"488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757\": rpc error: code = NotFound desc = could not find container \"488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757\": container with ID starting with 488402c5c55a9a4da63ade8dec23f16633ecacad15417a885d29cb1945d75757 not found: ID does not exist"
Sep 18 23:16:23.106260 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.106237    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"]
Sep 18 23:16:23.109215 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:23.109192    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4vt82"]
Sep 18 23:16:24.974574 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:16:24.974536    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da600b2-4065-42ed-a4e4-d388f49384f8" path="/var/lib/kubelet/pods/0da600b2-4065-42ed-a4e4-d388f49384f8/volumes"
Sep 18 23:18:53.289833 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:18:53.289797    2582 scope.go:117] "RemoveContainer" containerID="fb9338dc3536f67eb243543846c7bfc6c1c7a75e9374405fe0688601aed5f1f4"
Sep 18 23:20:45.555160 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:20:45.555129    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:20:45.561680 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:20:45.561653    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:20:45.575496 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:20:45.575474    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:20:45.581206 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:20:45.581187    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:25:45.604263 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:25:45.604233    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:25:45.610387 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:25:45.610363    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:25:45.625128 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:25:45.625106    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:25:45.630441 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:25:45.630421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:27:14.735633 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:14.735592    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-56b598698f-4h4fv_5318e729-a528-4255-87ad-f1080b6ffe7d/llm-d-routing-sidecar/0.log"
Sep 18 23:27:14.742526 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:14.742499    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-56b598698f-4h4fv_5318e729-a528-4255-87ad-f1080b6ffe7d/storage-initializer/0.log"
Sep 18 23:27:14.751033 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:14.751009    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-56b598698f-4h4fv_5318e729-a528-4255-87ad-f1080b6ffe7d/main/0.log"
Sep 18 23:27:16.029561 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:16.029528    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"]
Sep 18 23:27:31.053633 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.053565    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main" containerID="cri-o://3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e" gracePeriod=45
Sep 18 23:27:31.671190 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.671133    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="llm-d-routing-sidecar" containerID="cri-o://915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80" gracePeriod=45
Sep 18 23:27:31.705735 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.705705    2582 generic.go:356] "Generic (PLEG): container finished" podID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerID="3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e" exitCode=0
Sep 18 23:27:31.705855 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.705755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerDied","Data":"3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e"}
Sep 18 23:27:31.918026 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.917995    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:27:31.969032 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.968998    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjnfx\" (UniqueName: \"kubernetes.io/projected/5318e729-a528-4255-87ad-f1080b6ffe7d-kube-api-access-mjnfx\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969205 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-model-cache\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969205 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969064    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-home\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969205 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5318e729-a528-4255-87ad-f1080b6ffe7d-cabundle-cert\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969205 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969144    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-kserve-provision-location\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969441 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-tmp-dir\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969441 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969253    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-dshm\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969441 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5318e729-a528-4255-87ad-f1080b6ffe7d-tls-certs\") pod \"5318e729-a528-4255-87ad-f1080b6ffe7d\" (UID: \"5318e729-a528-4255-87ad-f1080b6ffe7d\") "
Sep 18 23:27:31.969441 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969323    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.969648 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969527    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5318e729-a528-4255-87ad-f1080b6ffe7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:27:31.969648 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969538    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:31.969972 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.969930    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-home" (OuterVolumeSpecName: "home") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.971510 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.971474    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5318e729-a528-4255-87ad-f1080b6ffe7d-kube-api-access-mjnfx" (OuterVolumeSpecName: "kube-api-access-mjnfx") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "kube-api-access-mjnfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:27:31.971632 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.971584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-dshm" (OuterVolumeSpecName: "dshm") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.971684 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.971653    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5318e729-a528-4255-87ad-f1080b6ffe7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:27:31.980859 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.980834    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:31.983578 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:31.983558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5318e729-a528-4255-87ad-f1080b6ffe7d" (UID: "5318e729-a528-4255-87ad-f1080b6ffe7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:27:32.070698 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.070671    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjnfx\" (UniqueName: \"kubernetes.io/projected/5318e729-a528-4255-87ad-f1080b6ffe7d-kube-api-access-mjnfx\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.070698 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.070695    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.070698 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.070704    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5318e729-a528-4255-87ad-f1080b6ffe7d-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.071126 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.070713    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.071126 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.070721    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.071126 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.070729    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5318e729-a528-4255-87ad-f1080b6ffe7d-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.071126 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.070737    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5318e729-a528-4255-87ad-f1080b6ffe7d-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:27:32.712763 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.712728    2582 generic.go:356] "Generic (PLEG): container finished" podID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerID="915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80" exitCode=0
Sep 18 23:27:32.712929 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.712797    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"
Sep 18 23:27:32.712929 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.712809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerDied","Data":"915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80"}
Sep 18 23:27:32.712929 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.712866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv" event={"ID":"5318e729-a528-4255-87ad-f1080b6ffe7d","Type":"ContainerDied","Data":"699aff9a6d32890999b550fc93b5a8a669946ff4ede372b593088403e340425e"}
Sep 18 23:27:32.712929 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.712894    2582 scope.go:117] "RemoveContainer" containerID="3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e"
Sep 18 23:27:32.722657 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.722638    2582 scope.go:117] "RemoveContainer" containerID="edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5"
Sep 18 23:27:32.730507 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.730490    2582 scope.go:117] "RemoveContainer" containerID="915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80"
Sep 18 23:27:32.736823 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.736803    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"]
Sep 18 23:27:32.739490 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.739472    2582 scope.go:117] "RemoveContainer" containerID="3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e"
Sep 18 23:27:32.739795 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:27:32.739774    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e\": container with ID starting with 3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e not found: ID does not exist" containerID="3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e"
Sep 18 23:27:32.739848 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.739805    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e"} err="failed to get container status \"3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e\": rpc error: code = NotFound desc = could not find container \"3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e\": container with ID starting with 3961f8ef0de57fd2867ee1e334678b5fd643712d266df629262188cbe468068e not found: ID does not exist"
Sep 18 23:27:32.739848 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.739823    2582 scope.go:117] "RemoveContainer" containerID="edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5"
Sep 18 23:27:32.740148 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:27:32.740127    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5\": container with ID starting with edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5 not found: ID does not exist" containerID="edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5"
Sep 18 23:27:32.740217 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.740156    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5"} err="failed to get container status \"edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5\": rpc error: code = NotFound desc = could not find container \"edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5\": container with ID starting with edd070b22f788e43322e1f3991878d06786db09e57a9edbba5da0e96a5ca0fc5 not found: ID does not exist"
Sep 18 23:27:32.740217 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.740171    2582 scope.go:117] "RemoveContainer" containerID="915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80"
Sep 18 23:27:32.740297 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.740216    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-56b598698f-4h4fv"]
Sep 18 23:27:32.740405 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:27:32.740389    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80\": container with ID starting with 915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80 not found: ID does not exist" containerID="915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80"
Sep 18 23:27:32.740442 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.740410    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80"} err="failed to get container status \"915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80\": rpc error: code = NotFound desc = could not find container \"915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80\": container with ID starting with 915555c69be5edc778af2a273dc578a5e0fd69523ecbd420f6630d8658866d80 not found: ID does not exist"
Sep 18 23:27:32.974111 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:27:32.974032    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" path="/var/lib/kubelet/pods/5318e729-a528-4255-87ad-f1080b6ffe7d/volumes"
Sep 18 23:28:31.422054 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422017    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"]
Sep 18 23:28:31.422619 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422554    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="storage-initializer"
Sep 18 23:28:31.422619 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422574    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="storage-initializer"
Sep 18 23:28:31.422619 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422594    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main"
Sep 18 23:28:31.422619 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422602    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main"
Sep 18 23:28:31.422619 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422611    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da600b2-4065-42ed-a4e4-d388f49384f8" containerName="main"
Sep 18 23:28:31.422619 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422618    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da600b2-4065-42ed-a4e4-d388f49384f8" containerName="main"
Sep 18 23:28:31.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422639    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="llm-d-routing-sidecar"
Sep 18 23:28:31.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422648    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="llm-d-routing-sidecar"
Sep 18 23:28:31.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422759    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="llm-d-routing-sidecar"
Sep 18 23:28:31.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422773    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5318e729-a528-4255-87ad-f1080b6ffe7d" containerName="main"
Sep 18 23:28:31.422972 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.422788    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da600b2-4065-42ed-a4e4-d388f49384f8" containerName="main"
Sep 18 23:28:31.426166 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.426141    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.428390 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.428370    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 23:28:31.428390 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.428382    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-zwj7d\""
Sep 18 23:28:31.428580 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.428371    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 23:28:31.428580 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.428493    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 23:28:31.428713 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.428697    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 23:28:31.428756 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.428700    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 23:28:31.437655 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.437632    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"]
Sep 18 23:28:31.467704 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-cabundle-cert\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.467893 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-dshm\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.467893 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-home\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.467893 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-model-cache\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.467893 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tls-certs\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.467893 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tmp-dir\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.468171 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kserve-provision-location\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.468171 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.467979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfrx\" (UniqueName: \"kubernetes.io/projected/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kube-api-access-qlfrx\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.568828 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.568791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-home\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569064 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.568837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-model-cache\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569064 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.568862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tls-certs\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569064 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.568890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tmp-dir\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569064 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.568920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kserve-provision-location\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569064 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.568980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfrx\" (UniqueName: \"kubernetes.io/projected/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kube-api-access-qlfrx\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569064 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.569056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-cabundle-cert\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569356 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.569087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-dshm\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569356 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.569281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-home\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569356 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.569304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-model-cache\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569611 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.569589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kserve-provision-location\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569681 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.569618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tmp-dir\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.569745 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.569728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-cabundle-cert\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.571391 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.571367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-dshm\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.571689 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.571672    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tls-certs\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.576456 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.576430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfrx\" (UniqueName: \"kubernetes.io/projected/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kube-api-access-qlfrx\") pod \"prestop-hook-test-kserve-767b458bf9-gvpmx\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:31.736714 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:31.736636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:32.017011 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:32.016973    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"]
Sep 18 23:28:32.017973 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:28:32.017921    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0fe99d_f89a_4b1c_b797_2cfb33a9cb61.slice/crio-93d51db721a8f248f1f870a851f7de7a16d08cf2f7ec39a405b6e722dba7fe2e WatchSource:0}: Error finding container 93d51db721a8f248f1f870a851f7de7a16d08cf2f7ec39a405b6e722dba7fe2e: Status 404 returned error can't find the container with id 93d51db721a8f248f1f870a851f7de7a16d08cf2f7ec39a405b6e722dba7fe2e
Sep 18 23:28:33.028551 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:33.028509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" event={"ID":"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61","Type":"ContainerStarted","Data":"ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b"}
Sep 18 23:28:33.029036 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:33.028572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" event={"ID":"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61","Type":"ContainerStarted","Data":"93d51db721a8f248f1f870a851f7de7a16d08cf2f7ec39a405b6e722dba7fe2e"}
Sep 18 23:28:35.042649 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:35.042612    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerID="ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b" exitCode=0
Sep 18 23:28:35.043065 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:35.042693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" event={"ID":"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61","Type":"ContainerDied","Data":"ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b"}
Sep 18 23:28:36.052002 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:36.051958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" event={"ID":"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61","Type":"ContainerStarted","Data":"e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6"}
Sep 18 23:28:36.075223 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:36.075170    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podStartSLOduration=2.879493963 podStartE2EDuration="5.07515396s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.195659997s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:28:36.072194767 +0000 UTC m=+5873.105373664" watchObservedRunningTime="2026-09-18 23:28:36.07515396 +0000 UTC m=+5873.108332853"
Sep 18 23:28:36.737597 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:36.737564    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:41.736878 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:41.736837    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:28:41.738646 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:41.738612    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:28:51.738006 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:28:51.737933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:29:01.737289 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:29:01.737236    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:29:11.737599 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:29:11.737538    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:29:21.737316 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:29:21.737270    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:29:31.737566 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:29:31.737468    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:29:41.738010 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:29:41.737968    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:29:51.737313 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:29:51.737270    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:30:01.738258 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:01.738207    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:30:11.737918 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:11.737869    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:30:21.737771 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:21.737723    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 18 23:30:31.747298 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:31.747264    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:30:31.755072 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:31.755043    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:30:36.678610 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:36.678570    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"]
Sep 18 23:30:45.642438 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:45.642410    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:30:45.648250 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:45.648228    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:30:45.664867 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:45.664849    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:30:45.670484 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:45.670463    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rmj6f_8e53a3c0-3b7d-4d05-9357-c7d53f7d096b/ovn-acl-logging/0.log"
Sep 18 23:30:51.701467 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:51.701407    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main" containerID="cri-o://e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6" gracePeriod=45
Sep 18 23:30:52.449333 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.449306    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:30:52.517048 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.516974    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-home\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517048 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-model-cache\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517048 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517035    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tls-certs\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517312 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517074    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kserve-provision-location\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517312 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517115    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlfrx\" (UniqueName: \"kubernetes.io/projected/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kube-api-access-qlfrx\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517312 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tmp-dir\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517312 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517191    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-cabundle-cert\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517312 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517242    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-dshm\") pod \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\" (UID: \"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61\") "
Sep 18 23:30:52.517312 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517291    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:30:52.517650 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517535    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-model-cache\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.517739 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517704    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-home" (OuterVolumeSpecName: "home") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:30:52.517866 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.517846    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 23:30:52.519599 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.519563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kube-api-access-qlfrx" (OuterVolumeSpecName: "kube-api-access-qlfrx") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "kube-api-access-qlfrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 23:30:52.519738 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.519707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-dshm" (OuterVolumeSpecName: "dshm") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:30:52.520077 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.520055    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 23:30:52.533440 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.533412    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:30:52.542289 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.542264    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" (UID: "bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 23:30:52.618839 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.618806    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kserve-provision-location\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.618839 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.618836    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlfrx\" (UniqueName: \"kubernetes.io/projected/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-kube-api-access-qlfrx\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.619029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.618850    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tmp-dir\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.619029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.618865    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-cabundle-cert\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.619029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.618875    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-dshm\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.619029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.618886    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-home\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.619029 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.618896    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61-tls-certs\") on node \"ip-10-0-142-209.ec2.internal\" DevicePath \"\""
Sep 18 23:30:52.836102 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.836004    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerID="e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6" exitCode=0
Sep 18 23:30:52.836102 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.836056    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"
Sep 18 23:30:52.836102 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.836082    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" event={"ID":"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61","Type":"ContainerDied","Data":"e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6"}
Sep 18 23:30:52.836698 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.836146    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx" event={"ID":"bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61","Type":"ContainerDied","Data":"93d51db721a8f248f1f870a851f7de7a16d08cf2f7ec39a405b6e722dba7fe2e"}
Sep 18 23:30:52.836698 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.836172    2582 scope.go:117] "RemoveContainer" containerID="e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6"
Sep 18 23:30:52.846710 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.846690    2582 scope.go:117] "RemoveContainer" containerID="ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b"
Sep 18 23:30:52.855207 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.855182    2582 scope.go:117] "RemoveContainer" containerID="e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6"
Sep 18 23:30:52.855532 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:30:52.855479    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6\": container with ID starting with e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6 not found: ID does not exist" containerID="e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6"
Sep 18 23:30:52.855532 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.855514    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6"} err="failed to get container status \"e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6\": rpc error: code = NotFound desc = could not find container \"e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6\": container with ID starting with e3fc8d6e406ff9d3d89b90180a87a74de1a5df99c053b38ec681b1c349b152e6 not found: ID does not exist"
Sep 18 23:30:52.855696 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.855538    2582 scope.go:117] "RemoveContainer" containerID="ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b"
Sep 18 23:30:52.856130 ip-10-0-142-209 kubenswrapper[2582]: E0918 23:30:52.856071    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b\": container with ID starting with ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b not found: ID does not exist" containerID="ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b"
Sep 18 23:30:52.856130 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.856107    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b"} err="failed to get container status \"ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b\": rpc error: code = NotFound desc = could not find container \"ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b\": container with ID starting with ed2a69409265651d6dbb94afda958facdbcccc4cebaaf6f0c55bb9355eb8904b not found: ID does not exist"
Sep 18 23:30:52.858708 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.858686    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"]
Sep 18 23:30:52.862939 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.862914    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-767b458bf9-gvpmx"]
Sep 18 23:30:52.974284 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:30:52.974243    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" path="/var/lib/kubelet/pods/bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61/volumes"
Sep 18 23:33:33.856361 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:33.856321    2582 ???:1] "http: TLS handshake error from 10.0.142.209:52744: EOF"
Sep 18 23:33:33.858940 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:33.858918    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-657b5f585b-96g68_3529e86f-3aea-4f0e-9f49-6250abcb042c/router/0.log"
Sep 18 23:33:34.626772 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:34.626742    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-657b5f585b-96g68_3529e86f-3aea-4f0e-9f49-6250abcb042c/router/0.log"
Sep 18 23:33:35.358701 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:35.358677    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-657b5f585b-96g68_3529e86f-3aea-4f0e-9f49-6250abcb042c/router/0.log"
Sep 18 23:33:36.083969 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:36.083919    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-657b5f585b-96g68_3529e86f-3aea-4f0e-9f49-6250abcb042c/router/0.log"
Sep 18 23:33:36.999000 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:36.998971    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:38.160358 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:38.160329    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:39.678227 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:39.678202    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:40.940083 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:40.940056    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:42.068781 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:42.068753    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:43.138335 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:43.138306    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:44.219663 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:44.219628    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:45.289022 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:45.288989    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:46.333922 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:46.333893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:47.510106 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:47.510075    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:48.595154 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:48.595129    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:49.646013 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:49.645986    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:50.739013 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:50.738985    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:51.861030 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:51.861000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:52.903645 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:52.903617    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:54.012517 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:54.012490    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:55.068538 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:55.068514    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:56.113058 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:56.113032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:57.138913 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:57.138843    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:58.206365 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:58.206337    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:33:59.242585 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:33:59.242555    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:00.366316 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:00.366283    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:01.433819 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:01.433790    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:02.509429 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:02.509401    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:03.577075 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:03.577046    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:04.612601 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:04.612574    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:05.705974 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:05.705924    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:06.736304 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:06.736276    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:07.785144 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:07.785116    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:08.977729 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:08.977697    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:10.055094 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:10.055067    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:11.120836 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:11.120806    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:12.188868 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:12.188831    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:13.211309 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:13.211278    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:14.233398 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:14.233367    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:15.255368 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:15.255337    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:16.279997 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:16.279967    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:18.535386 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.535350    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6w2r/must-gather-f7kkf"]
Sep 18 23:34:18.535800 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.535729    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="storage-initializer"
Sep 18 23:34:18.535800 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.535744    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="storage-initializer"
Sep 18 23:34:18.535800 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.535760    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main"
Sep 18 23:34:18.535800 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.535765    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main"
Sep 18 23:34:18.536023 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.535853    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0fe99d-f89a-4b1c-b797-2cfb33a9cb61" containerName="main"
Sep 18 23:34:18.540745 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.540715    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:18.543720 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.543694    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6w2r\"/\"kube-root-ca.crt\""
Sep 18 23:34:18.544063 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.544041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6w2r\"/\"openshift-service-ca.crt\""
Sep 18 23:34:18.548151 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.548125    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/must-gather-f7kkf"]
Sep 18 23:34:18.703337 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.703304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/82c1c02e-1559-4cf0-ace3-a69eb91a35fb-must-gather-output\") pod \"must-gather-f7kkf\" (UID: \"82c1c02e-1559-4cf0-ace3-a69eb91a35fb\") " pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:18.703575 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.703363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbl7r\" (UniqueName: \"kubernetes.io/projected/82c1c02e-1559-4cf0-ace3-a69eb91a35fb-kube-api-access-rbl7r\") pod \"must-gather-f7kkf\" (UID: \"82c1c02e-1559-4cf0-ace3-a69eb91a35fb\") " pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:18.804534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.804427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbl7r\" (UniqueName: \"kubernetes.io/projected/82c1c02e-1559-4cf0-ace3-a69eb91a35fb-kube-api-access-rbl7r\") pod \"must-gather-f7kkf\" (UID: \"82c1c02e-1559-4cf0-ace3-a69eb91a35fb\") " pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:18.804534 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.804521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/82c1c02e-1559-4cf0-ace3-a69eb91a35fb-must-gather-output\") pod \"must-gather-f7kkf\" (UID: \"82c1c02e-1559-4cf0-ace3-a69eb91a35fb\") " pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:18.804904 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.804881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/82c1c02e-1559-4cf0-ace3-a69eb91a35fb-must-gather-output\") pod \"must-gather-f7kkf\" (UID: \"82c1c02e-1559-4cf0-ace3-a69eb91a35fb\") " pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:18.813180 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.813160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbl7r\" (UniqueName: \"kubernetes.io/projected/82c1c02e-1559-4cf0-ace3-a69eb91a35fb-kube-api-access-rbl7r\") pod \"must-gather-f7kkf\" (UID: \"82c1c02e-1559-4cf0-ace3-a69eb91a35fb\") " pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:18.853436 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:18.853409    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/must-gather-f7kkf"
Sep 18 23:34:19.124039 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:19.124011    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/must-gather-f7kkf"]
Sep 18 23:34:19.126326 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:34:19.126285    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82c1c02e_1559_4cf0_ace3_a69eb91a35fb.slice/crio-1f2b1c03ce0c69f158b53e6b7437b6958d806897e08630bc13f50e2bed7986ea WatchSource:0}: Error finding container 1f2b1c03ce0c69f158b53e6b7437b6958d806897e08630bc13f50e2bed7986ea: Status 404 returned error can't find the container with id 1f2b1c03ce0c69f158b53e6b7437b6958d806897e08630bc13f50e2bed7986ea
Sep 18 23:34:19.128075 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:19.128055    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 23:34:19.909165 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:19.909126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/must-gather-f7kkf" event={"ID":"82c1c02e-1559-4cf0-ace3-a69eb91a35fb","Type":"ContainerStarted","Data":"1f2b1c03ce0c69f158b53e6b7437b6958d806897e08630bc13f50e2bed7986ea"}
Sep 18 23:34:20.920012 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:20.919971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/must-gather-f7kkf" event={"ID":"82c1c02e-1559-4cf0-ace3-a69eb91a35fb","Type":"ContainerStarted","Data":"ca89ce5563be9ce7434692220d5596c8773166432d4dbfe3d507fe56d5fbabf1"}
Sep 18 23:34:20.921670 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:20.921638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/must-gather-f7kkf" event={"ID":"82c1c02e-1559-4cf0-ace3-a69eb91a35fb","Type":"ContainerStarted","Data":"4215cb5788044ac650cfb5009337b244d9ba93af8c3f9b3d987a8d5d8a45bcb4"}
Sep 18 23:34:20.940063 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:20.940000    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6w2r/must-gather-f7kkf" podStartSLOduration=2.149419101 podStartE2EDuration="2.93997737s" totalImagesPullingTime="790.558269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 23:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:34:20.93922693 +0000 UTC m=+6217.972405825" watchObservedRunningTime="2026-09-18 23:34:20.93997737 +0000 UTC m=+6217.973156266"
Sep 18 23:34:21.637758 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:21.637728    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gddqh_299a1b6f-acaa-4fe6-a60f-0ef9f999970f/global-pull-secret-syncer/0.log"
Sep 18 23:34:21.786396 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:21.786367    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-976mf_8b132a4a-79f6-4d95-a440-7962162b5a24/konnectivity-agent/0.log"
Sep 18 23:34:21.912636 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:21.912530    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-209.ec2.internal_6e9479c83aec32841bba93774c695096/haproxy/0.log"
Sep 18 23:34:27.232507 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:27.232479    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-csmlv_be78e7a8-a0a1-46ab-8765-d0b0c9aefaf9/kuadrant-console-plugin/0.log"
Sep 18 23:34:28.969676 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:28.969646    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-cb4bk_15aebc2b-a06e-4c9f-ac52-24c8158b2f62/kube-state-metrics/0.log"
Sep 18 23:34:28.997721 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:28.997689    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-cb4bk_15aebc2b-a06e-4c9f-ac52-24c8158b2f62/kube-rbac-proxy-main/0.log"
Sep 18 23:34:29.020853 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.020818    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-cb4bk_15aebc2b-a06e-4c9f-ac52-24c8158b2f62/kube-rbac-proxy-self/0.log"
Sep 18 23:34:29.090738 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.090701    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-492lt_702652c4-e8c4-4692-bcbc-14728c18b1e4/monitoring-plugin/0.log"
Sep 18 23:34:29.145068 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.145014    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9p8rv_1e49dc96-baa9-43b8-8567-9584ad189e2c/node-exporter/0.log"
Sep 18 23:34:29.178437 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.178240    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9p8rv_1e49dc96-baa9-43b8-8567-9584ad189e2c/kube-rbac-proxy/0.log"
Sep 18 23:34:29.219672 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.219613    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9p8rv_1e49dc96-baa9-43b8-8567-9584ad189e2c/init-textfile/0.log"
Sep 18 23:34:29.427016 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.426983    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qhv2n_da788116-d78b-428f-9b38-133b7e5105f5/kube-rbac-proxy-main/0.log"
Sep 18 23:34:29.453167 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.453062    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qhv2n_da788116-d78b-428f-9b38-133b7e5105f5/kube-rbac-proxy-self/0.log"
Sep 18 23:34:29.477689 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.477606    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qhv2n_da788116-d78b-428f-9b38-133b7e5105f5/openshift-state-metrics/0.log"
Sep 18 23:34:29.907292 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.907256    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c9d7f8c77-5p257_6d47c0e8-a685-4bc9-a1a1-68e770448194/thanos-query/0.log"
Sep 18 23:34:29.928852 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.928822    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c9d7f8c77-5p257_6d47c0e8-a685-4bc9-a1a1-68e770448194/kube-rbac-proxy-web/0.log"
Sep 18 23:34:29.955634 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.955609    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c9d7f8c77-5p257_6d47c0e8-a685-4bc9-a1a1-68e770448194/kube-rbac-proxy/0.log"
Sep 18 23:34:29.978579 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:29.978550    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c9d7f8c77-5p257_6d47c0e8-a685-4bc9-a1a1-68e770448194/prom-label-proxy/0.log"
Sep 18 23:34:30.001250 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:30.001211    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c9d7f8c77-5p257_6d47c0e8-a685-4bc9-a1a1-68e770448194/kube-rbac-proxy-rules/0.log"
Sep 18 23:34:30.036633 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:30.036601    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c9d7f8c77-5p257_6d47c0e8-a685-4bc9-a1a1-68e770448194/kube-rbac-proxy-metrics/0.log"
Sep 18 23:34:31.460317 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.460277    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"]
Sep 18 23:34:31.468132 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.468097    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.472961 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.472916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"]
Sep 18 23:34:31.473325 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.473303    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s6w2r\"/\"default-dockercfg-hpdwh\""
Sep 18 23:34:31.540185 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.540133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-podres\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.540185 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.540188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-proc\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.540436 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.540312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-sys\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.540436 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.540359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6n22\" (UniqueName: \"kubernetes.io/projected/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-kube-api-access-g6n22\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.540436 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.540392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-lib-modules\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.641276 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.641231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-lib-modules\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.641579 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.641549    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-lib-modules\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.641745 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.641582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-podres\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.641907 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.641887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-proc\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.642209 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.642188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-sys\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.642442 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.642416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6n22\" (UniqueName: \"kubernetes.io/projected/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-kube-api-access-g6n22\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.642520 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.641700    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-podres\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.642520 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.642363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-sys\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.642520 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.642079    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-proc\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.651497 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.651464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6n22\" (UniqueName: \"kubernetes.io/projected/5bb3a5fc-a162-4f36-ab3f-70f8da6fd520-kube-api-access-g6n22\") pod \"perf-node-gather-daemonset-ng8wq\" (UID: \"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520\") " pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:31.777980 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.777842    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/2.log"
Sep 18 23:34:31.783475 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.783443    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-q4dqp_67298389-9b58-4a22-8f3d-610f24790eb1/console-operator/3.log"
Sep 18 23:34:31.785118 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:31.785076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:32.078892 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:32.078840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"]
Sep 18 23:34:32.080671 ip-10-0-142-209 kubenswrapper[2582]: W0918 23:34:32.080627    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5bb3a5fc_a162_4f36_ab3f_70f8da6fd520.slice/crio-aceb21b85febdbc318ce2f20875b657abac22e3edf76b710fd939282d035b59c WatchSource:0}: Error finding container aceb21b85febdbc318ce2f20875b657abac22e3edf76b710fd939282d035b59c: Status 404 returned error can't find the container with id aceb21b85febdbc318ce2f20875b657abac22e3edf76b710fd939282d035b59c
Sep 18 23:34:32.336261 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:32.336162    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-bgdjw_99f0f008-3991-4448-9772-bf0266a9919e/download-server/0.log"
Sep 18 23:34:32.949799 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:32.949768    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-2rkn8_e73cb2a4-8da4-45ef-8aca-7e784ce00dd4/volume-data-source-validator/0.log"
Sep 18 23:34:33.014877 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:33.014836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq" event={"ID":"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520","Type":"ContainerStarted","Data":"f82ada65d32dd91224f9fccdc4525c4b483c65f27316efa61398b7643cce3f26"}
Sep 18 23:34:33.015092 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:33.014905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq" event={"ID":"5bb3a5fc-a162-4f36-ab3f-70f8da6fd520","Type":"ContainerStarted","Data":"aceb21b85febdbc318ce2f20875b657abac22e3edf76b710fd939282d035b59c"}
Sep 18 23:34:33.015092 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:33.015074    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:33.052848 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:33.052777    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq" podStartSLOduration=2.052755692 podStartE2EDuration="2.052755692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 23:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 23:34:33.04740134 +0000 UTC m=+6230.080580236" watchObservedRunningTime="2026-09-18 23:34:33.052755692 +0000 UTC m=+6230.085934587"
Sep 18 23:34:34.005126 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:34.005094    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4tmv_9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c/dns/0.log"
Sep 18 23:34:34.025776 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:34.025744    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4tmv_9ebd9ea5-1ea3-46c5-b03b-4c0224a1b68c/kube-rbac-proxy/0.log"
Sep 18 23:34:34.138912 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:34.138885    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2wm8s_310ea0c0-e1ce-4eb1-871f-33ef46f48827/dns-node-resolver/0.log"
Sep 18 23:34:34.717909 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:34.717881    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4kct_6e2d73b3-52ef-4f05-839a-051bab482a3c/node-ca/0.log"
Sep 18 23:34:35.896811 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:35.896781    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-657b5f585b-96g68_3529e86f-3aea-4f0e-9f49-6250abcb042c/router/0.log"
Sep 18 23:34:36.479816 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:36.479785    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wjwjl_bc847e3d-5072-4c7d-9ae0-f8fc928ab82f/serve-healthcheck-canary/0.log"
Sep 18 23:34:37.041563 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:37.041533    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tv5mn_e3838c43-2bb5-4a08-bde1-9844e7401f3a/kube-rbac-proxy/0.log"
Sep 18 23:34:37.068307 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:37.068281    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tv5mn_e3838c43-2bb5-4a08-bde1-9844e7401f3a/exporter/0.log"
Sep 18 23:34:37.094541 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:37.094510    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tv5mn_e3838c43-2bb5-4a08-bde1-9844e7401f3a/extractor/0.log"
Sep 18 23:34:39.035319 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:39.035288    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s6w2r/perf-node-gather-daemonset-ng8wq"
Sep 18 23:34:40.347041 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:40.347007    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7fdf8f455-crjpm_bb800afb-8362-493c-a4c4-ba434637a7bd/manager/0.log"
Sep 18 23:34:41.569622 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:41.569595    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-657b5f585b-96g68_3529e86f-3aea-4f0e-9f49-6250abcb042c/router/0.log"
Sep 18 23:34:42.396535 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:42.396494    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-kv7l2_42449c59-91d2-4d2a-93aa-efb5a91da3f1/manager/0.log"
Sep 18 23:34:42.435088 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:42.435058    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9jbt8_21daa3ca-74a0-4198-b928-75d738e85c9c/s3-init/0.log"
Sep 18 23:34:42.461630 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:42.461595    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9jbt8_21daa3ca-74a0-4198-b928-75d738e85c9c/download-hf-model/0.log"
Sep 18 23:34:42.492807 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:42.492778    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-8fqmw_d6a16208-ebef-4fce-982c-43e7551359b4/seaweedfs/0.log"
Sep 18 23:34:42.753449 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:42.753364    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-657b5f585b-96g68_3529e86f-3aea-4f0e-9f49-6250abcb042c/router/0.log"
Sep 18 23:34:47.800709 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:47.800670    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-vl67p_3e494d22-0a87-4169-ac16-40037cdeaee6/kube-storage-version-migrator-operator/1.log"
Sep 18 23:34:47.802546 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:47.802521    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-vl67p_3e494d22-0a87-4169-ac16-40037cdeaee6/kube-storage-version-migrator-operator/0.log"
Sep 18 23:34:48.986454 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:48.986423    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-76p9s_7c8563dd-0e2e-4aab-8814-c44cce27c793/kube-multus/0.log"
Sep 18 23:34:49.169483 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:49.169447    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fn6vl_847de099-1e25-441a-8dff-e4d9378896bd/kube-multus-additional-cni-plugins/0.log"
Sep 18 23:34:49.190248 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:49.190214    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fn6vl_847de099-1e25-441a-8dff-e4d9378896bd/egress-router-binary-copy/0.log"
Sep 18 23:34:49.209896 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:49.209870    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fn6vl_847de099-1e25-441a-8dff-e4d9378896bd/cni-plugins/0.log"
Sep 18 23:34:49.231644 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:49.231620    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fn6vl_847de099-1e25-441a-8dff-e4d9378896bd/bond-cni-plugin/0.log"
Sep 18 23:34:49.253110 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:49.253012    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fn6vl_847de099-1e25-441a-8dff-e4d9378896bd/routeoverride-cni/0.log"
Sep 18 23:34:49.274411 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:49.274387    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fn6vl_847de099-1e25-441a-8dff-e4d9378896bd/whereabouts-cni-bincopy/0.log"
Sep 18 23:34:49.295368 ip-10-0-142-209 kubenswrapper[2582]: I0918 23:34:49.295342    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fn6vl_847de099-1e25-441a-8dff-e4d9378896bd/whereabouts-cni/0.log"